2,438 packages matching “tokenizer”

tokenizer

v1.1.2 · 13 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

A wide purpose tokenizer for node.js which looks like a stream

1.9KDownloads across all versions in the last 7 days, from the official npm downloads API.

No known vulnerabilities

Known vulnerabilities affecting v1.1.2 (the latest release), from the OSV.dev database.

js-tokens

v10.0.0 · 9 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Tiny JavaScript tokenizer.

185.2MDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v10.0.0 (the latest release), from the OSV.dev database.

strtok3

v10.3.5 · 5 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

A promise based streaming tokenizer

24.2MDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v10.3.5 (the latest release), from the OSV.dev database.

leac

v0.7.0 · 1 year ago

95
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Lexer / tokenizer

12.5MDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.7.0 (the latest release), from the OSV.dev database.

@jscpd/tokenizer

v4.2.6 · 1 month ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

tokenizer of source code for jscpd

858.3KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v4.2.6 (the latest release), from the OSV.dev database.

@anthropic-ai/tokenizer

v0.0.4 · 3 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Claude tokenizer

204.7KDownloads across all versions in the last 7 days, from the official npm downloads API.Apache-2.0License declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.0.4 (the latest release), from the OSV.dev database.

@tokenizer/range

v0.13.1 · 7 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Range-request tokenizer adapter

24.7KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.13.1 (the latest release), from the OSV.dev database.

@tokenizer/s3

v1.0.4 · 4 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Amazon S3 tokenizer

21.7KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.0.4 (the latest release), from the OSV.dev database.

gpt-tokenizer

v4.0.0 · 1 month ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

A pure JavaScript BPE tokenizer with model metadata and pricing for GPT-5, GPT-4o, o-series, and other OpenAI models

1.2MDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v4.0.0 (the latest release), from the OSV.dev database.

ai-tokenizer

v1.0.6 · 9 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

A faster than tiktoken tokenizer with first-class support for Vercel's AI SDK.

79.8KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.0.6 (the latest release), from the OSV.dev database.

wink-tokenizer

v5.3.0 · 4 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Multilingual tokenizer that automatically tags each token with its type

52.3KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v5.3.0 (the latest release), from the OSV.dev database.

@shikijs/primitive

v4.4.3 · 1 month ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Tokenizer primitives of Shiki

9.2MDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v4.4.3 (the latest release), from the OSV.dev database.

@yozora/tokenizer-image

v2.4.1 · 2 days ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

<!-- :begin use tokenizer/banner -->

7.0KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v2.4.1 (the latest release), from the OSV.dev database.

@yozora/tokenizer-list

v2.4.1 · 2 days ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

<!-- :begin use tokenizer/banner -->

7.0KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v2.4.1 (the latest release), from the OSV.dev database.

moo

v0.5.3 · 6 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Optimised tokenizer/lexer generator! 🐄 Much performance. Moo!

8.4MDownloads across all versions in the last 7 days, from the official npm downloads API.BSD-3-ClauseLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.5.3 (the latest release), from the OSV.dev database.

@yozora/tokenizer-admonition

v2.4.1 · 2 days ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Tokenizer for processing admonitions

6.4KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v2.4.1 (the latest release), from the OSV.dev database.

@yozora/tokenizer-paragraph

v2.4.1 · 2 days ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

<!-- :begin use tokenizer/banner -->

7.0KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v2.4.1 (the latest release), from the OSV.dev database.

@yozora/tokenizer-delete

v2.4.1 · 2 days ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

<!-- :begin use tokenizer/banner -->

7.0KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v2.4.1 (the latest release), from the OSV.dev database.

simple-html-tokenizer

v0.5.11 · 5 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.

439.2KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.5.11 (the latest release), from the OSV.dev database.

@humanwhocodes/momoa

v3.3.13 · 13 days ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

JSON AST parser, tokenizer, printer, traverser.

2.4MDownloads across all versions in the last 7 days, from the official npm downloads API.Apache-2.0License declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v3.3.13 (the latest release), from the OSV.dev database.
1 / 50Next