A Language Independent 'word finding' tool, useful for stemming, tokenizing, indexing, spell checking and other common NLP tasks. Works on any human language and any unicode character set, learns from the data you give it. (Uses compression, maximum entro

Virtual, zero-copy strings for AssemblyScript - slice, trim and search without allocating

A lightweight, zero-dependency JavaScript library for adding markdown formatting functionality to any textarea element

A language server for alphaTab providing coding assistance for alphaTex.

Tokenize JavaScript

[](https://github.com/jonluca/gpt4-tokenizer-utils/actions/workflows/main.yml) [](https://ww

Bedrock Tokenization Engine

Light-weight tool for normalizing whitespace, splitting lines, and accurately tokenizing words (no regex). Multiple natural languages supported.

Server-side syntax-highlighting helper for Pi-Studio. A pure-JavaScript regex tokeniser — no external highlighting library, and (deliberately) **no runtime dependencies at all**.

Modern WebAssembly distribution of sudachi.rs

String Tokenizer for Node.js using ICU's BreakIterators

Property path-to-value resolver, in TypeScript

Advanced tagging/tokenizing plugin for input fields with a focus on keyboard and copy-paste support.

Tests for CSS Tokenizers

Uses snapdragon to tokenize a single JavaScript block comment into an object, with description, tags, and code example sections that can be passed to any other comment parsers for further parsing.

Bedrock Tokenization Engine

Normalize and clean text

A Bedrock module to provide auto-rotating tokenizers

a ported version of gTTS python library which is a interface to Google's Text to Speech API

Token-efficiency linter for LLM prompts and payloads