56 packages matching “tokeniser”

tokeniser

v0.0.3 · 9 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Dynamic file contents manipulation in node.js utilising inline token strings.

12Downloads across all versions in the last 7 days, from the official npm downloads API.ISCLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.0.3 (the latest release), from the OSV.dev database.

tiktoken-node

v0.0.7 · 2 years ago

90
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

tiktoken is a fast [BPE](https://en.wikipedia.org/wiki/Byte_pair_encoding) tokeniser for use with OpenAI's models.

7.7KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.0.7 (the latest release), from the OSV.dev database.

string-tokeniser

v2.0.3 · 2 years ago

90
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

A simple string tokeniser

24Downloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v2.0.3 (the latest release), from the OSV.dev database.

taibun

v1.1.3 · 2 years ago

90
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Taiwanese Hokkien Transliterator and Tokeniser

37Downloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.1.3 (the latest release), from the OSV.dev database.

@av-pi-studio/highlight

v0.0.59 · 59 minutes ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Server-side syntax-highlighting helper for Pi-Studio. A pure-JavaScript regex tokeniser — no external highlighting library, and (deliberately) **no runtime dependencies at all**.

2.7KDownloads across all versions in the last 7 days, from the official npm downloads API.

No known vulnerabilities

Known vulnerabilities affecting v0.0.59 (the latest release), from the OSV.dev database.

escuelle

v0.0.11 · 7 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

SQL tokeniser & parser

5Downloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.0.11 (the latest release), from the OSV.dev database.

irctokens

v2.0.1 · 7 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

RFC1459 and IRCv3 protocol tokeniser

10Downloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v2.0.1 (the latest release), from the OSV.dev database.

@benjamin_r/yalt

v0.1.0 · 3 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Yet Another LaTeX Tokeniser - a zero-dependency streaming LaTeX tokeniser built for LLM text output.

5Downloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.1.0 (the latest release), from the OSV.dev database.

@mrlowlevel/tiktoken-node

v0.0.8 · 3 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

tiktoken is a fast [BPE](https://en.wikipedia.org/wiki/Byte_pair_encoding) tokeniser for use with OpenAI's models.

4Downloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.0.8 (the latest release), from the OSV.dev database.

@kindly-note/render-markdown

v0.1.0 · 2 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

One-call markdown → safe HTML for kindly-note. renderMarkdown(src, opts) composes @kindly-note/lang-markdown (CommonMark tokeniser) + @kindly-note/emitters-markdown (semantic-HTML emitter with security-first defaults) + a sensible Highlighter, with option

1.0KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.1.0 (the latest release), from the OSV.dev database.

tiktoken-nodeless

v0.0.8 · 3 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

tiktoken is a fast [BPE](https://en.wikipedia.org/wiki/Byte_pair_encoding) tokeniser for use with OpenAI's models.

1Downloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.0.8 (the latest release), from the OSV.dev database.

expressionparser

v1.1.9 · 1 year ago

95
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Parse simple expressions, in a language of your own description

8.5KDownloads across all versions in the last 7 days, from the official npm downloads API.BSD-2-ClauseLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.1.9 (the latest release), from the OSV.dev database.

tiktoken

v1.0.22 · 11 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

JS/WASM bindings for tiktoken

1.5MDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.0.22 (the latest release), from the OSV.dev database.

cborg

v6.1.1 · 4 days ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Fast CBOR with a focus on strictness

272.2KDownloads across all versions in the last 7 days, from the official npm downloads API.Apache-2.0License declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v6.1.1 (the latest release), from the OSV.dev database.

@txm/tokeniser

v1.1.0 · 10 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

search and replace tokens from a source file, writing to a new file

7Downloads across all versions in the last 7 days, from the official npm downloads API.ISCLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.1.0 (the latest release), from the OSV.dev database.

@dqbd/tiktoken

v1.0.22 · 11 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

JS/WASM bindings for tiktoken

374.2KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.0.22 (the latest release), from the OSV.dev database.

js-tokeniser

v1.1.2 · 8 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Tokenizes a text from left to right and returns the list of tokens with their types.

6Downloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.1.2 (the latest release), from the OSV.dev database.

js-tiktoken

v1.0.21 · 11 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

JavaScript port of tiktoken

6.7MDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.0.21 (the latest release), from the OSV.dev database.

html-tokeniser

v0.0.2 · 3 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

HTML tokeniser that supports streaming interfaces. Converts HTML into tokens and provides options to transforma and convert those tokens back into a HTML string.

1Downloads across all versions in the last 7 days, from the official npm downloads API.ISCLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.0.2 (the latest release), from the OSV.dev database.

tokeniser-package

v1.0.3 · 11 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

A simple tokenizer for encoding/decoding text into numeric tokens.

7Downloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.0.3 (the latest release), from the OSV.dev database.
1 / 3Next