The 1/3 million most frequent words, all lowercase, with counts.

Fast tokenizer for language models, supporting BPE, Unigram and WordPiece tokenization

VerbX Localize — offline ONNX translation: Unigram tokenizer + encoder/decoder generation, no network.

Node.js binding for huggingface/tokenizers library

Word prediction for T9 keyboard.

Mailwoman neural-classifier weights for locale 'en-us'. Data-only package — loaded by @mailwoman/neural at runtime.

A natural language model and cross-language model, for natural language understanding and generation

Mailwoman neural classifier runtime: SentencePiece tokenizer + ONNX inference + decoder wiring.

Browser LLM inference on WebGPU using Three.js TSL compute shaders.

Store and apply business rules as structured per-day Markdown files with BM25 search

A safe, searchable, approval-gated Markdown knowledge vault for DeepSeek Harness.

Mailwoman neural-classifier weights for locale 'fr-fr'. Data-only overlay — shares the base model.onnx + tokenizer.model from @mailwoman/neural-weights-en-us (they are byte-identical); ships only the fr-specific data siblings. Loaded by @mailwoman/neural

Scoped, bounded, approval-gated long-term memory for DeepSeek Harness agents.

This repository holds the code for the TokenGeeX Rust crate and Python package. TokenGeeX is a tokenizer for [CodeGeeX](https://github.com/THUDM/Codegeex2) aimed at code and Chinese. It is based on [UnigramLM (Taku Kudo 2018)](https://arxiv.org/abs/1804.1

The fastest way to start training your own Language Model. Create production-ready LLM training projects in seconds.

The fastest way to start training your own Language Model. Create production-ready LLM training projects in seconds.

Fast greedy pattern discovery and lattice-backed tokenizer for sequential data.

A package for building and analyzing word nGrams

Convert Korean historical source XML (국사편찬위원회 한국사DB) into a searchable SQLite BM25 corpus — hanja-primary full-text search, structured place/person index, co-occurrence clustering, and an optional LLM literal-translation secondary index.

Estimate and count tokens (incl. exact OpenAI BPE) and input costs for LLM API calls