Fast token estimation with 95%+ average accuracy in a 2kB bundle

r/w stream of glsl tokens

tokenizer of source code for jscpd

Common token types for decoding and encoding numeric and string values

Tokenize a shell string into argv array

ProseMirror Markdown integration

A faster than tiktoken tokenizer with first-class support for Vercel's AI SDK.

Claude tokenizer

Multi-arch builds of HuggingFace tokenizers

JS tokenizer for Mistral-based LLMs

gemma3 tokenizer for NodeJS/Browser

detector of copy/paste in files

Parse CSS media query lists.

core functionality of copy/paste detector for jscpd

A primitive to tokenize your solid-components to enable custom parsing.

Parse CSS Cascade Layer names.

JavaScript implementation of Japanese morphological analyzer

Range-request tokenizer adapter

safely inject a block of tokens into a shader

Tokenizer for processing fenced math block (formulas)