Small library that provides functions to tokenize a string into an array of words with or without punctuation

TypeScript definitions for wink-tokenizer

<!-- :begin use tokenizer/banner -->

Optimised tokenizer/lexer generator! 🐄 Much performance. Moo!

deepseek_v3 tokenizer for NodeJS/Browser

Renderer for kramed outputting plain text that can easily be fed to a search indexer

Retrieve the values defined with preprocessor statements in a selection of GLSL tokens

Determine the scope depth of an array of GLSL tokens

This [remark][remark] plugin can disable any or all remark `blockTokenizers` and `inlineTokenizers`. It can not only disable the ones provided by remark core, but also any other tokenizer that has been added to the remark parser whether through plugins or

Fork of @lexical/code with Prism.js syntax highlighting removed. Drop-in replacement for @lexical/code.

"Descope" an array of GLSL tokens such that they can be safely inlined alongside within another shader without causing any global variable conflicts.

Infer the scope of each token in an array of GLSL tokens

Error-tolerant ECMAScript parser

Our library `@lenml/llama2-tokenizer` has been deprecated. We are excited to introduce our new library `@lenml/tokenizers` as its replacement, offering a broader set of features and an enhanced experience.

Wix Restaurants credit-cards tokenizer

qwen3 tokenizer for NodeJS/Browser

TS tokenizer for Mistral-based LLMs

A simple tool to generate bert tokens and input features

Stop-words for Orama

Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.