A primitive to tokenize your solid-components to enable custom parsing.

Parse CSS Cascade Layer names.

JavaScript implementation of Japanese morphological analyzer

Range-request tokenizer adapter

A text tokenizer module for converting blocks of text into keywords

safely inject a block of tokens into a shader

Tokenizer for processing fenced math block (formulas)

Tokenizer utilities for Tokenlens.

Takes an array of GLSL tokens and determines whether or not they're a property of another identifier

CSS / Extended CSS tokenizer

Trim the whitespace within an array of GLSL tokens

Parse CSS color values

Tokenizer for processing inline math (formulas)

Tokenizer for OpenAI large language models.

Shows code error fragment of input file

Tokenizer for processing admonitions

Provide a high level wrapper for kuromoji.js

My JavaScript parser

JS tokenizer for LLaMA 3

React component to convert a text string into React.ReactNodes