Optimised tokenizer/lexer generator! 🐄 Much performance. Moo!

Convert into a lower case with spaces between words, then capitalize text

deepseek_v3 tokenizer for NodeJS/Browser

Retrieve the values defined with preprocessor statements in a selection of GLSL tokens

Determine the scope depth of an array of GLSL tokens

This [remark][remark] plugin can disable any or all remark `blockTokenizers` and `inlineTokenizers`. It can not only disable the ones provided by remark core, but also any other tokenizer that has been added to the remark parser whether through plugins or

[](https://github.com/jonluca/gpt4-tokenizer-utils/actions/workflows/main.yml) [](https://ww

"Descope" an array of GLSL tokens such that they can be safely inlined alongside within another shader without causing any global variable conflicts.

Infer the scope of each token in an array of GLSL tokens

Escape carriage return the right way.

Wix Restaurants credit-cards tokenizer

qwen3 tokenizer for NodeJS/Browser

Provide a high level wrapper for kuromoji.js

TS tokenizer for Mistral-based LLMs

Convert SQL statements into a list of tokens

Simple algorithm to tokenize Chinese texts into words using CC-CEDICT.

gemma2 tokenizer for NodeJS/Browser

HTML and CSS lexer aimed at code with fatal errors, accepts mixed coding languages

TypeScript definitions for css-selector-tokenizer

Type-safe APIs with Protobuf and TypeScript.