328,637 packages matching “Text tokenizer”

@textlint-rule/tokenize-text

v2.0.2 · 3 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Javascript text tokenizer that is easy to use and compose

236Downloads across all versions in the last 7 days, from the official npm downloads API.Apache-2.0License declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v2.0.2 (the latest release), from the OSV.dev database.

tokenize-text

v1.1.3 · 10 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Javascript text tokenizer that is easy to use and compose

94Downloads across all versions in the last 7 days, from the official npm downloads API.Apache-2.0License declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.1.3 (the latest release), from the OSV.dev database.

cjk-tokenizer

v0.1.0 · 9 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

A CJK text tokenizer

25Downloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.1.0 (the latest release), from the OSV.dev database.

@csstools/css-tokenizer

v4.0.0 · 7 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Tokenize CSS

59.8MDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v4.0.0 (the latest release), from the OSV.dev database.

strtok3

v10.3.5 · 5 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

A promise based streaming tokenizer

25.7MDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v10.3.5 (the latest release), from the OSV.dev database.

@tokenizer/inflate

v0.4.1 · 9 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Tokenized zip support

25.3MDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.4.1 (the latest release), from the OSV.dev database.

@tokenizer/token

v0.3.0 · 5 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

TypeScript definition for strtok3 token

27.4MDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.3.0 (the latest release), from the OSV.dev database.

gpt-tokenizer

v4.0.0 · 6 days ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

A pure JavaScript BPE tokenizer with model metadata and pricing for GPT-5, GPT-4o, o-series, and other OpenAI models

1.4MDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v4.0.0 (the latest release), from the OSV.dev database.

tokenizer.js

v1.3.2 · 8 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

A modern runtime text tokenizer, for modern web.

3Downloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.3.2 (the latest release), from the OSV.dev database.

social-text-tokenizer

v3.2.3 · 4 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Posts/comments text parser for FreeFeed.net

55Downloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v3.2.3 (the latest release), from the OSV.dev database.

simple-html-tokenizer

v0.5.11 · 5 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.

580.5KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.5.11 (the latest release), from the OSV.dev database.

text-tokenizer

v1.0.4 · 7 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

A simple perfomant tokenizer

63Downloads across all versions in the last 7 days, from the official npm downloads API.ISCLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.0.4 (the latest release), from the OSV.dev database.

tokenx

v2.1.0 · 17 days ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Fast token estimation with 95%+ average accuracy in a 2kB bundle

1.5MDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v2.1.0 (the latest release), from the OSV.dev database.

prosemirror-markdown

v1.13.6 · 6 days ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

ProseMirror Markdown integration

9.3MDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.13.6 (the latest release), from the OSV.dev database.

scss-tokenizer

v0.4.3 · 4 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

A tokenzier for Sass' SCSS syntax

816.8KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.4.3 (the latest release), from the OSV.dev database.

@anthropic-ai/tokenizer

v0.0.4 · 3 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Claude tokenizer

278.7KDownloads across all versions in the last 7 days, from the official npm downloads API.Apache-2.0License declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.0.4 (the latest release), from the OSV.dev database.

charabia-js

v0.2.0 · 1 year ago

95
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

A WebAssembly binding for the charabia multilingual text tokenizer used by Meilisearch.

12Downloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.2.0 (the latest release), from the OSV.dev database.

@csstools/css-parser-algorithms

v4.0.0 · 7 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Algorithms to help you parse CSS from an array of tokens.

59.5MDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v4.0.0 (the latest release), from the OSV.dev database.

search-text-tokenizer

v2.1.1 · 9 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

A tokenizer for Google-like search queries

43Downloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v2.1.1 (the latest release), from the OSV.dev database.

ret

v0.5.0 · 3 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Tokenizes a string that represents a regular expression.

37.0MDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.5.0 (the latest release), from the OSV.dev database.
1 / 50Next