328,318 packages matching “text-tokenizer”

@csstools/css-color-parser

v4.2.0 · 7 days ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Parse CSS color values

55.0MDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v4.2.0 (the latest release), from the OSV.dev database.

@microsoft/tiktokenizer

v1.0.10 · 1 year ago

95
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Tokenizer for OpenAI large language models.

79.2KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.0.10 (the latest release), from the OSV.dev database.

code-error-fragment

v0.0.230 · 8 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Shows code error fragment of input file

571.4KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.0.230 (the latest release), from the OSV.dev database.

kuromojin

v3.0.1 · 1 year ago

95
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Provide a high level wrapper for kuromoji.js

233.5KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v3.0.1 (the latest release), from the OSV.dev database.

html-tokenizer

v4.0.1 · 5 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Small, fast, event-driven, fault-tolerant html tokenizer. Works in node or browsers.

4.5KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v4.0.1 (the latest release), from the OSV.dev database.

@lenml/tokenizer-gemma3

v3.7.2 · 11 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

gemma3 tokenizer for NodeJS/Browser

21.4KDownloads across all versions in the last 7 days, from the official npm downloads API.Apache-2.0License declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v3.7.2 (the latest release), from the OSV.dev database.

@lenml/tokenizers

v3.7.2 · 11 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

a lightweight no-dependency fork of transformers.js (only tokenizers)

55.6KDownloads across all versions in the last 7 days, from the official npm downloads API.Apache-2.0License declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v3.7.2 (the latest release), from the OSV.dev database.

zeparser

v0.0.7 · 13 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

My JavaScript parser

44.8KDownloads across all versions in the last 7 days, from the official npm downloads API.

No known vulnerabilities

Known vulnerabilities affecting v0.0.7 (the latest release), from the OSV.dev database.

llama-tokenizer-js

v1.2.2 · 2 years ago

90
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

JS tokenizer for LLaMA-based LLMs

20.2KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.2.2 (the latest release), from the OSV.dev database.

@anush008/tokenizers

v0.6.0 · 5 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Multi-arch builds of HuggingFace tokenizers

117.5KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.6.0 (the latest release), from the OSV.dev database.

gpt4-tokenizer

v1.3.1 · 1 month ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

[![Build](https://github.com/jonluca/gpt4-tokenizer-utils/actions/workflows/main.yml/badge.svg)](https://github.com/jonluca/gpt4-tokenizer-utils/actions/workflows/main.yml) [![NPM Version](https://img.shields.io/npm/v/gpt4-tokenizer-utils.svg)](https://ww

1.7KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.3.1 (the latest release), from the OSV.dev database.

leac

v0.7.0 · 11 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Lexer / tokenizer

10.7MDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.7.0 (the latest release), from the OSV.dev database.

@tokenizer/range

v0.13.1 · 6 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Range-request tokenizer adapter

10.5KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.13.1 (the latest release), from the OSV.dev database.

vue-eslint-parser-template-tokenizer-pug

v1.0.0 · 1 year ago

95
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

This is an internal library. If you just want to lint your pug templates, use [eslint-plugin-vue-pug](https://github.com/rashfael/eslint-plugin-vue-pug).

24.2KDownloads across all versions in the last 7 days, from the official npm downloads API.ISCLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.0.0 (the latest release), from the OSV.dev database.

@yozora/core-tokenizer

v2.4.0 · 21 days ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

<header> <h1 align="center"> <a href="https://github.com/yozorajs/yozora/tree/v2.3.17/packages/core-tokenizer#readme">@yozora/core-tokenizer</a> </h1> <div align="center"> <a href="https://www.npmjs.com/package/@yozora/core-tokenizer">

10.3KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v2.4.0 (the latest release), from the OSV.dev database.

react-text-tokenizer

v1.0.4 · 1 year ago

95
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

React component to convert a text string into React.ReactNodes

3Downloads across all versions in the last 7 days, from the official npm downloads API.ISCLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.0.4 (the latest release), from the OSV.dev database.

sbd

v1.0.19 · 5 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Split text into sentences with Sentence Boundary Detection (SBD).

49.0KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.0.19 (the latest release), from the OSV.dev database.

@yozora/tokenizer-paragraph

v2.4.0 · 21 days ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

<!-- :begin use tokenizer/banner -->

10.3KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v2.4.0 (the latest release), from the OSV.dev database.

@yozora/tokenizer-delete

v2.4.0 · 21 days ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

<!-- :begin use tokenizer/banner -->

10.3KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v2.4.0 (the latest release), from the OSV.dev database.

@yozora/tokenizer-heading

v2.4.0 · 21 days ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

<!-- :begin use tokenizer/banner -->

10.3KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v2.4.0 (the latest release), from the OSV.dev database.
Prev3 / 50Next