A collection of tools is designed to aid in working with the Rumantsch (Romansh) language

Stacks Artificial Intelligence.

through stream that translates glsl-parser AST nodes into working glsl code

Fast tokenizer for language models, supporting BPE, Unigram and WordPiece tokenization

Node.js binding for huggingface/tokenizers library

A minimalistic JavaScript implementation of the Jinja templating engine, specifically designed for parsing and rendering ML chat templates.

Advanced html to plain text converter

Extract a shader's name from its source

Token-aware AI prompt compression — same results, fewer tokens. Works with every LLM.

Capacitor 8 plugin for Conekta card tokenization - create payment tokens from card details via direct API calls

Accurate and fast sentiment scoring of phrases with emoticons :) & emojis 🎉

React-based typeahead and typeahead-tokenizer

lezer-based Julia grammar

JSON AST parser, tokenizer, printer, traverser.

Tokenometer CLI — LLM token cost + latency benchmarking across Claude, GPT-4o, Gemini, Mistral, and Cohere. Multi-format, empirical mode, vision tokens, SARIF output.

Streaming Markdoc parser, component registry, and system prompt generator for LLM generative UI

Simple CSS tokenizer

Tokenizer of c-like languages.

parse me!

TypeScript port of Sudachi morphological analyzer for Japanese text