A fastify plugin to add Tokenize support through a decorator.

Stream Transform class to tokenize RTF, and another to de-encapsulate text or HTML

Tokenize paragraphs into sentences, and smaller tokens.

Simple algorithm to tokenize Chinese texts into words using CC-CEDICT.

Small library that provides functions to tokenize a string into an array of words with or without punctuation

Simple way to tokenize cards with Malga

basic tokenize function

Tokenize AEON source text for parsers, editors, and tooling.

Turns a string into tokens.

Tokenize CSS according to the CSS Syntax

printf format string tokenize

Simple CSS Parser to tokenize CSS, merge rules, and optimize it

Tokenize source code documentation

Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.

Client-side Paysafe adapter for PayFanout (Paysafe.js hosted iframe fields, tokenize-first). No secrets, no server code.

Read a file, tokenize it, and spit out a handy JSON.

Tokenize a search query with refined options (like Google's) and match against a target

Client-side PayPal adapter for PayFanout (PayPal Buttons, popup approval, tokenize-first). No secrets, no server code.

A NodeJS library to tokenize strings

personal tokenize util