328,657 packages matching “Text tokenizer”

sql-tokenizer

v1.0.5 · 10 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Convert SQL statements into a list of tokens

1.2KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.0.5 (the latest release), from the OSV.dev database.

chinese-tokenizer

v2.4.0 · 7 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Simple algorithm to tokenize Chinese texts into words using CC-CEDICT.

976Downloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v2.4.0 (the latest release), from the OSV.dev database.

@plurnk/plurnk-mimetypes-tokenizers

v1.8.0 · 21 hours ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Bundled LLM tokenizer vocabularies (tokenizer.json set) for @plurnk/plurnk-mimetypes' tokenizer seam — exact token counting for window math.

729Downloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.8.0 (the latest release), from the OSV.dev database.

@lenml/tokenizer-gemma2

v3.7.2 · 11 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

gemma2 tokenizer for NodeJS/Browser

2.7KDownloads across all versions in the last 7 days, from the official npm downloads API.Apache-2.0License declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v3.7.2 (the latest release), from the OSV.dev database.

codsen-tokenizer

v7.0.25 · 2 years ago

90
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

HTML and CSS lexer aimed at code with fatal errors, accepts mixed coding languages

690Downloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v7.0.25 (the latest release), from the OSV.dev database.

@meistrari/gpt-tokenizer

v0.3.2 · 3 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 / Claude Instant / Claude 2

528Downloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.3.2 (the latest release), from the OSV.dev database.

@bpinternal/thicktoken

v3.1.0 · 1 month ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

A bundled cl100k tokenizer (Rust→WASM) with count/truncate/split helpers

8.2KDownloads across all versions in the last 7 days, from the official npm downloads API.ISCLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v3.1.0 (the latest release), from the OSV.dev database.

@types/css-selector-tokenizer

v0.7.4 · 2 years ago

90
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

TypeScript definitions for css-selector-tokenizer

3.3KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.7.4 (the latest release), from the OSV.dev database.

@lenml/tokenizer-mistral_nemo

v3.7.2 · 11 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

mistral_nemo tokenizer for NodeJS/Browser

701Downloads across all versions in the last 7 days, from the official npm downloads API.Apache-2.0License declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v3.7.2 (the latest release), from the OSV.dev database.

@lenml/tokenizer-llama2

v3.7.2 · 11 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

llama2 tokenizer for NodeJS/Browser

638Downloads across all versions in the last 7 days, from the official npm downloads API.Apache-2.0License declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v3.7.2 (the latest release), from the OSV.dev database.

@rmenke/css-tokenizer-tests

v1.2.0 · 1 year ago

95
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Tests for CSS Tokenizers

2.8KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.2.0 (the latest release), from the OSV.dev database.

@croct/json5-parser

v0.2.2 · 8 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

A lossless JSON5 tokenizer and parser for Node.js that maintains indentation, spacing, and comments.

595.4KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.2.2 (the latest release), from the OSV.dev database.

glsl-token-functions

v1.0.1 · 9 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Extract function definitions from an array of GLSL tokens.

29.9KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.0.1 (the latest release), from the OSV.dev database.

@yozora/core-parser

v2.4.0 · 21 days ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

<header> <h1 align="center"> <a href="https://github.com/yozorajs/yozora/tree/v2.3.17/packages/core-parser#readme">@yozora/core-parser</a> </h1> <div align="center"> <a href="https://www.npmjs.com/package/@yozora/core-parser"> <img

10.3KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v2.4.0 (the latest release), from the OSV.dev database.

@stacksjs/ts-css

v0.1.4 · 3 days ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Pure-TypeScript CSS toolkit for Bun/Node — parser, walker, generator, selector engine, and minifier. Zero runtime deps.

88.9KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.1.4 (the latest release), from the OSV.dev database.

sentence-tokenizer

v1.0.1 · 7 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Tokenize paragraphs into sentences, and smaller tokens.

436Downloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.0.1 (the latest release), from the OSV.dev database.

@lenml/tokenizer-llama3

v3.7.2 · 11 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

llama3 tokenizer for NodeJS/Browser

262Downloads across all versions in the last 7 days, from the official npm downloads API.Apache-2.0License declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v3.7.2 (the latest release), from the OSV.dev database.

garu-ko

v0.9.16 · 1 day ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Ultra-lightweight Korean morphological analyzer for the web (1.2MB model, WASM, F1 96.0% gold testset)

1.2KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.9.16 (the latest release), from the OSV.dev database.

simple-text-tokenizer

v1.0.1 · 5 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Library to tokenize text to paragraphs, sentences, subsentences and words

4Downloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.0.1 (the latest release), from the OSV.dev database.

@orama/stemmers

v3.1.18 · 8 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Stemmers for Orama

34.1KDownloads across all versions in the last 7 days, from the official npm downloads API.Apache-2.0License declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v3.1.18 (the latest release), from the OSV.dev database.
Prev7 / 50Next