A robust HTML entities encoder/decoder with full Unicode support.

Our library `@lenml/llama2-tokenizer` has been deprecated. We are excited to introduce our new library `@lenml/tokenizers` as its replacement, offering a broader set of features and an enhanced experience.

A Markdown-to-typed-data parser built from scratch on the plgg framework: a layout-marker frontmatter splitter and a block tokenizer for the plggpress subset that produce an immutable Box-union AST (Result, never throw)

A streaming tokenizer

The Astropay Tokenizer SDK is a tool to tokenize and reveal sensitive payment information

Zero-dep streaming markdown for the browser. Rust→WASM core, Web Worker per stream, incremental parse with speculative closure.

A HTML encoder/decoder with full Unicode support.

Diff and markup HTML with <ins> and <del> tags

Vensim grammar built in ANTLR 4

Bivo Payments

A fast Lua lexer written in TypeScript

A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 / Claude Instant / Claude 2

language-server implementation for openVALIDATION

A Bedrock module to provide auto-rotating tokenizers

Pure card collection module for PayGood

A library which provides functionality for reducing and processing words and phrases.

A robust HTML entities encoder/decoder with full Unicode support.

Smart contracts to convert PENDLE to xPENDLE

Tokenize string input into 32-bit integers (for full text search and natural language processing).

Transform @ mentions into markdown links.