Trim the whitespace within an array of GLSL tokens

a lightweight no-dependency fork of transformers.js (only tokenizers)

This [remark][remark] plugin can disable any or all remark `blockTokenizers` and `inlineTokenizers`. It can not only disable the ones provided by remark core, but also any other tokenizer that has been added to the remark parser whether through plugins or

Retrieve the values defined with preprocessor statements in a selection of GLSL tokens

Determine the scope depth of an array of GLSL tokens

"Descope" an array of GLSL tokens such that they can be safely inlined alongside within another shader without causing any global variable conflicts.

Infer the scope of each token in an array of GLSL tokens

Multi-arch builds of HuggingFace tokenizers

claude tokenizer for NodeJS/Browser

gemini tokenizer for NodeJS/Browser

Provide a high level wrapper for kuromoji.js

Small, fast, event-driven, fault-tolerant html tokenizer. Works in node or browsers.

Forked version of kuromoji with better compatibility for browsers

TypeScript definitions for css-selector-tokenizer

deepseek_v3 tokenizer for NodeJS/Browser

<header> <h1 align="center"> <a href="https://github.com/yozorajs/yozora/tree/v2.3.17/packages/core-parser#readme">@yozora/core-parser</a> </h1> <div align="center"> <a href="https://www.npmjs.com/package/@yozora/core-parser"> <img

Tests for CSS Tokenizers

Extract function definitions from an array of GLSL tokens.

Fork of @lexical/code with Prism.js syntax highlighting removed. Drop-in replacement for @lexical/code.

Pure-TypeScript CSS toolkit for Bun/Node — parser, walker, generator, selector engine, and minifier. Zero runtime deps.