TextEncoder and TextDecoder APIs from Encoding Standard APIs in a universal package

Stream Transform class to tokenize RTF, and another to de-encapsulate text or HTML

Truncate text for React.js

Parse, validate, traverse, transform, and optimize Oniguruma regular expressions

Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.

An infinite scroll directive for vue.js.

Utility to fix TypeScript declarations when using default exports in CommonJS.

Convert cron expressions into human readable descriptions

A light-weight module that brings Fetch API to node.js

Convert SQL statements into a list of tokens

Simple algorithm to tokenize Chinese texts into words using CC-CEDICT.

Bundled LLM tokenizer vocabularies (tokenizer.json set) for @plurnk/plurnk-mimetypes' tokenizer seam — exact token counting for window math.

gemma2 tokenizer for NodeJS/Browser

HTML and CSS lexer aimed at code with fatal errors, accepts mixed coding languages

The assert module from Node.js, for the browser.

A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 / Claude Instant / Claude 2

Extend hammer.js with event propagation

Node.js library for Portable Executable format

Detect the indentation of code (commonjs fork)

A bundled cl100k tokenizer (Rust→WASM) with count/truncate/split helpers