1,731 packages matching “porifa-tokenizer”
@jscpd/finder
v4.3.0 · 6 days ago
detector of copy/paste in files
No known vulnerabilities
args-tokenizer
v0.3.0 · 1 year ago
Tokenize a shell string into argv array
No known vulnerabilities
@csstools/media-query-list-parser
v5.0.0 · 7 months ago
Parse CSS media query lists.
No known vulnerabilities
@jscpd/core
v4.2.5 · 2 months ago
core functionality of copy/paste detector for jscpd
No known vulnerabilities
js-tokens
v10.0.0 · 8 months ago
Tiny JavaScript tokenizer.
No known vulnerabilities
@solid-primitives/jsx-tokenizer
v1.1.4 · 1 month ago
A primitive to tokenize your solid-components to enable custom parsing.
No known vulnerabilities
@csstools/cascade-layer-name-parser
v3.0.0 · 7 months ago
Parse CSS Cascade Layer names.
No known vulnerabilities
gpt3-tokenizer
v1.1.5 · 3 years ago
[](https://github.com/botisan-ai/gpt3-tokenizer/actions/workflows/main.yml) [](https://www.npmjs.com/
No known vulnerabilities
@csstools/css-color-parser
v4.2.0 · 4 days ago
Parse CSS color values
No known vulnerabilities
glsl-token-properties
v1.0.1 · 11 years ago
Takes an array of GLSL tokens and determines whether or not they're a property of another identifier
No known vulnerabilities
@microsoft/tiktokenizer
v1.0.10 · 1 year ago
Tokenizer for OpenAI large language models.
No known vulnerabilities
code-error-fragment
v0.0.230 · 8 years ago
Shows code error fragment of input file
No known vulnerabilities
@huggingface/tokenizers
v0.1.3 · 5 months ago
🤗 Tokenizers.js: A pure JS/TS implementation of today's most used tokenizers
No known vulnerabilities
@lenml/tokenizer-gemma3
v3.7.2 · 11 months ago
gemma3 tokenizer for NodeJS/Browser
No known vulnerabilities
llama-tokenizer-js
v1.2.2 · 2 years ago
JS tokenizer for LLaMA-based LLMs
No known vulnerabilities
zeparser
v0.0.7 · 13 years ago
My JavaScript parser
No known vulnerabilities
@tokenizer/range
v0.13.1 · 6 months ago
Range-request tokenizer adapter
No known vulnerabilities
vue-eslint-parser-template-tokenizer-pug
v1.0.0 · 1 year ago
This is an internal library. If you just want to lint your pug templates, use [eslint-plugin-vue-pug](https://github.com/rashfael/eslint-plugin-vue-pug).
No known vulnerabilities
@yozora/core-tokenizer
v2.4.0 · 19 days ago
<header> <h1 align="center"> <a href="https://github.com/yozorajs/yozora/tree/v2.3.17/packages/core-tokenizer#readme">@yozora/core-tokenizer</a> </h1> <div align="center"> <a href="https://www.npmjs.com/package/@yozora/core-tokenizer">
No known vulnerabilities
@yozora/tokenizer-paragraph
v2.4.0 · 19 days ago
<!-- :begin use tokenizer/banner -->
No known vulnerabilities