328,656 packages matching “Text tokenizer”

@tokenizer/http

v0.9.2 · 1 year ago

95
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

HTTP tokenizer for Node.js and browser

987Downloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.9.2 (the latest release), from the OSV.dev database.

llm-cost

v1.0.5 · 2 years ago

90
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

[![Tests](https://github.com/rogeriochaves/llm-cost/actions/workflows/node.js.yml/badge.svg)](https://github.com/rogeriochaves/llm-cost/actions/workflows/node.js.yml) [![npm version](https://badge.fury.io/js/llm-cost.svg)](https://www.npmjs.com/package/ll

7.4KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.0.5 (the latest release), from the OSV.dev database.

tokenizer

v1.1.2 · 13 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

A wide purpose tokenizer for node.js which looks like a stream

1.7KDownloads across all versions in the last 7 days, from the official npm downloads API.

No known vulnerabilities

Known vulnerabilities affecting v1.1.2 (the latest release), from the OSV.dev database.

@sutton-signwriting/core

v2.0.1 · 9 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

a javascript package for node and browsers that supports general processing of the Sutton SignWriting script

6.7KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v2.0.1 (the latest release), from the OSV.dev database.

fast-tokenizer

v1.9.0 · 7 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

A fast tokenizer/lexer for JavaScript

38Downloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.9.0 (the latest release), from the OSV.dev database.

parser

v0.1.4 · 13 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

A configurable parser to parse whatever you want

1.8KDownloads across all versions in the last 7 days, from the official npm downloads API.

No known vulnerabilities

Known vulnerabilities affecting v0.1.4 (the latest release), from the OSV.dev database.

gemini-token-estimator

v0.6.0 · 1 year ago

95
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Estimate the number of tokens for Gemini models

3.7KDownloads across all versions in the last 7 days, from the official npm downloads API.ISCLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.6.0 (the latest release), from the OSV.dev database.

complexion

v0.2.2 · 5 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

A fast tokenizer/lexer core that is well tested.

1.6KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.2.2 (the latest release), from the OSV.dev database.

glsl-token-assignments

v2.0.2 · 9 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Take an array of GLSL tokens and determine which tokens are either assignments or variable declarations.

661.9KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v2.0.2 (the latest release), from the OSV.dev database.

@lenml/tokenizer-gpt35turbo

v3.7.2 · 11 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

gpt35turbo tokenizer for NodeJS/Browser

291Downloads across all versions in the last 7 days, from the official npm downloads API.Apache-2.0License declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v3.7.2 (the latest release), from the OSV.dev database.

tokenize-this

v1.4.2 · 6 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Turns a string into tokens.

15.5KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.4.2 (the latest release), from the OSV.dev database.

@curiousdannii/search-text-tokenizer

v2.0.0 · 10 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

A tokenizer for Google-like search queries

4Downloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v2.0.0 (the latest release), from the OSV.dev database.

@desertnet/scanner

v1.1.1 · 8 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

A regex-based string scanner/tokenizer

4.9KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.1.1 (the latest release), from the OSV.dev database.

@flowaccount/node-icu-tokenizer

v2.0.0 · 2 years ago

90
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

String Tokenizer for Node.js using ICU's BreakIterators

77Downloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v2.0.0 (the latest release), from the OSV.dev database.

text-tokenizer-utils

v0.0.3 · 3 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

A class Tokenizer to convert text documents into sequences of tokens

3Downloads across all versions in the last 7 days, from the official npm downloads API.Apache-2.0License declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.0.3 (the latest release), from the OSV.dev database.

rtf-stream-parser

v4.0.0 · 11 days ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Stream Transform class to tokenize RTF, and another to de-encapsulate text or HTML

57.5KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v4.0.0 (the latest release), from the OSV.dev database.

js-tokenizer

v1.3.3 · 12 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

No description provided.

1.3KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.3.3 (the latest release), from the OSV.dev database.

@astropay/tokenizer-sdk

v1.1.3 · 1 year ago

95
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Official AstroPay client for easy frontend integration to Tokenizer

218Downloads across all versions in the last 7 days, from the official npm downloads API.UNLICENSEDLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.1.3 (the latest release), from the OSV.dev database.

@chonkiejs/token

v0.0.4 · 1 month ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

HuggingFace tokenizer support for Chonkie - extends @chonkiejs/core with real tokenization

265Downloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.0.4 (the latest release), from the OSV.dev database.

@lordluceus/mistral-tokenizer

v1.2.2 · 2 years ago

90
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

TS tokenizer for Mistral-based LLMs

287Downloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.2.2 (the latest release), from the OSV.dev database.
Prev8 / 50Next