363,986 packages matching “CommonJS text tokenizer”

args-tokenizer

v0.3.0 · 1 year ago

95
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Tokenize a shell string into argv array

149.8KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.3.0 (the latest release), from the OSV.dev database.

wink-tokenizer

v5.3.0 · 4 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Multilingual tokenizer that automatically tags each token with its type

74.9KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v5.3.0 (the latest release), from the OSV.dev database.

@yozora/tokenizer-text

v2.4.0 · 21 days ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

<!-- :begin use tokenizer/banner -->

8.8KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v2.4.0 (the latest release), from the OSV.dev database.

@huggingface/tokenizers

v0.1.3 · 5 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

🤗 Tokenizers.js: A pure JS/TS implementation of today's most used tokenizers

1.3MDownloads across all versions in the last 7 days, from the official npm downloads API.Apache-2.0License declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.1.3 (the latest release), from the OSV.dev database.

ansi-color

v0.2.2 · 8 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

This module provides basic ANSI color code support, to allow you to format your console output with foreground and background colors as well as providing bold, italic and underline support.

786.5KDownloads across all versions in the last 7 days, from the official npm downloads API.BSD LicenseLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.2.2 (the latest release), from the OSV.dev database.

ai-tokenizer

v1.0.6 · 8 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

A faster than tiktoken tokenizer with first-class support for Vercel's AI SDK.

133.9KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.0.6 (the latest release), from the OSV.dev database.

fs-extra

v11.4.0 · 29 days ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

fs-extra contains methods that aren't included in the vanilla Node.js fs package. Such as recursive mkdir, copy, and remove.

177.6MDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v11.4.0 (the latest release), from the OSV.dev database.

leac

v0.7.0 · 11 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Lexer / tokenizer

10.7MDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.7.0 (the latest release), from the OSV.dev database.

gpt3-tokenizer

v1.1.5 · 3 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

[![Build](https://github.com/botisan-ai/gpt3-tokenizer/actions/workflows/main.yml/badge.svg)](https://github.com/botisan-ai/gpt3-tokenizer/actions/workflows/main.yml) [![NPM Version](https://img.shields.io/npm/v/gpt3-tokenizer.svg)](https://www.npmjs.com/

22.3KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.1.5 (the latest release), from the OSV.dev database.

@babel/plugin-transform-modules-commonjs

v8.0.1 · 2 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

This plugin transforms ES2015 modules to CommonJS

64.5MDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v8.0.1 (the latest release), from the OSV.dev database.

@lenml/tokenizers

v3.7.2 · 11 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

a lightweight no-dependency fork of transformers.js (only tokenizers)

55.6KDownloads across all versions in the last 7 days, from the official npm downloads API.Apache-2.0License declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v3.7.2 (the latest release), from the OSV.dev database.

@lenml/tokenizer-gemma3

v3.7.2 · 11 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

gemma3 tokenizer for NodeJS/Browser

21.4KDownloads across all versions in the last 7 days, from the official npm downloads API.Apache-2.0License declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v3.7.2 (the latest release), from the OSV.dev database.

diff

v9.0.0 · 4 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

A JavaScript text diff implementation.

118.8MDownloads across all versions in the last 7 days, from the official npm downloads API.BSD-3-ClauseLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v9.0.0 (the latest release), from the OSV.dev database.

@jscpd/finder

v4.3.0 · 9 days ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

detector of copy/paste in files

973.1KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v4.3.0 (the latest release), from the OSV.dev database.

html-tokenizer

v4.0.1 · 5 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Small, fast, event-driven, fault-tolerant html tokenizer. Works in node or browsers.

4.5KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v4.0.1 (the latest release), from the OSV.dev database.

@csstools/media-query-list-parser

v5.0.0 · 7 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Parse CSS media query lists.

11.7MDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v5.0.0 (the latest release), from the OSV.dev database.

reflect-metadata

v0.2.2 · 2 years ago

90
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Polyfill for Metadata Reflection API

33.5MDownloads across all versions in the last 7 days, from the official npm downloads API.Apache-2.0License declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.2.2 (the latest release), from the OSV.dev database.

@jscpd/core

v4.2.5 · 2 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

core functionality of copy/paste detector for jscpd

1.2MDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v4.2.5 (the latest release), from the OSV.dev database.

@solid-primitives/jsx-tokenizer

v1.1.4 · 1 month ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

A primitive to tokenize your solid-components to enable custom parsing.

45.7KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.1.4 (the latest release), from the OSV.dev database.

@csstools/cascade-layer-name-parser

v3.0.0 · 7 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Parse CSS Cascade Layer names.

4.2MDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v3.0.0 (the latest release), from the OSV.dev database.
Prev2 / 50Next