Determine the scope depth of an array of GLSL tokens

This [remark][remark] plugin can disable any or all remark `blockTokenizers` and `inlineTokenizers`. It can not only disable the ones provided by remark core, but also any other tokenizer that has been added to the remark parser whether through plugins or

Fork of @lexical/code with Prism.js syntax highlighting removed. Drop-in replacement for @lexical/code.

"Descope" an array of GLSL tokens such that they can be safely inlined alongside within another shader without causing any global variable conflicts.

Infer the scope of each token in an array of GLSL tokens

claude tokenizer for NodeJS/Browser

Error-tolerant ECMAScript parser

Our library `@lenml/llama2-tokenizer` has been deprecated. We are excited to introduce our new library `@lenml/tokenizers` as its replacement, offering a broader set of features and an enhanced experience.

gemini tokenizer for NodeJS/Browser

JS tokenizer for Mistral-based LLMs

The API for building streaming tokenizers and lexers.

Stop-words for Orama

Small library that provides functions to tokenize a string into an array of words with or without punctuation

A bundled cl100k tokenizer (Rust→WASM) with count/truncate/split helpers

TypeScript definitions for css-selector-tokenizer

Forked version of kuromoji with better compatibility for browsers

deepseek_v3 tokenizer for NodeJS/Browser

Tests for CSS Tokenizers

Extract function definitions from an array of GLSL tokens.

<header> <h1 align="center"> <a href="https://github.com/yozorajs/yozora/tree/v2.3.17/packages/core-parser#readme">@yozora/core-parser</a> </h1> <div align="center"> <a href="https://www.npmjs.com/package/@yozora/core-parser"> <img