text-segmentation ==============

List of text file extensions

The fastest way to start training your own Language Model. Create production-ready LLM training projects in seconds.

gpt4 tokenizer for NodeJS/Browser

through stream that transforms glsl-parser AST nodes and rewrites variables into shorter forms

Text Decoder

The node core libs for in browser usage.

finite state machine tokenizer

Streaming text decoder that preserves multibyte Unicode characters

micromark extension to support definition lists

DeepSeek tokenizer families for tokkit.

Rampart — client-side PII redaction for AI assistants: deterministic recognizers + a 14.7 MB ONNX classifier (transformers.js), default-deny policy, and reversible placeholders. Runs entirely in the browser.

Tokenization utilities - HuggingFace tokenizers wrapper for NLPTools

Zero-dependency JSON serializer and parser built from scratch — matches native JSON.stringify/JSON.parse behavior, including the weird parts.

BPE tokenizer used for the NovelAI frontend.

The fastest tokenizer on the web. Exact token ids for any BPE or SentencePiece vocabulary, in browsers and Node.

Convert Chinese text to list of Chinese words

This package contains rich text helpers for Lexical.

(eventually) spec-compliant html 5 parser

BlockChyp Typescript Client