62 packages matching “tokeniser”

tokeniser

v0.0.3 · 10 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Dynamic file contents manipulation in node.js utilising inline token strings.

8Downloads across all versions in the last 7 days, from the official npm downloads API.ISCLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.0.3 (the latest release), from the OSV.dev database.

tiktoken-node

v0.0.7 · 2 years ago

90
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

tiktoken is a fast [BPE](https://en.wikipedia.org/wiki/Byte_pair_encoding) tokeniser for use with OpenAI's models.

5.5KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.0.7 (the latest release), from the OSV.dev database.

taibun

v1.1.3 · 2 years ago

90
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Taiwanese Hokkien Transliterator and Tokeniser

4Downloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.1.3 (the latest release), from the OSV.dev database.

string-tokeniser

v2.0.3 · 2 years ago

90
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

A simple string tokeniser

3Downloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v2.0.3 (the latest release), from the OSV.dev database.

dwc-gcode-core

v1.23.0 · 13 hours ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

RepRapFirmware-faithful G-code parsing for DuetWebControl plugins: an allocation-light line tokeniser, parameter parsing and in-place rewriting, conditional-G-code (if/var/while/...) recognition, a line-preserving config.g/macro directive editor, firmware

853Downloads across all versions in the last 7 days, from the official npm downloads API.GPL-3.0-or-laterLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.23.0 (the latest release), from the OSV.dev database.

escuelle

v0.0.11 · 7 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

SQL tokeniser & parser

5Downloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.0.11 (the latest release), from the OSV.dev database.

@benjamin_r/yalt

v0.1.0 · 5 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Yet Another LaTeX Tokeniser - a zero-dependency streaming LaTeX tokeniser built for LLM text output.

2Downloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.1.0 (the latest release), from the OSV.dev database.

irctokens

v2.0.1 · 9 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

RFC1459 and IRCv3 protocol tokeniser

6Downloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v2.0.1 (the latest release), from the OSV.dev database.

tiktoken

v1.0.22 · 1 year ago

95
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

JS/WASM bindings for tiktoken

1.1MDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.0.22 (the latest release), from the OSV.dev database.

cborg

v6.1.2 · 23 days ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Fast CBOR with a focus on strictness

275.9KDownloads across all versions in the last 7 days, from the official npm downloads API.Apache-2.0License declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v6.1.2 (the latest release), from the OSV.dev database.

expressionparser

v1.1.9 · 1 year ago

95
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Parse simple expressions, in a language of your own description

5.3KDownloads across all versions in the last 7 days, from the official npm downloads API.BSD-2-ClauseLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.1.9 (the latest release), from the OSV.dev database.

@dqbd/tiktoken

v1.0.22 · 1 year ago

95
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

JS/WASM bindings for tiktoken

266.7KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.0.22 (the latest release), from the OSV.dev database.

@kindly-note/render-markdown

v0.1.0 · 4 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

One-call markdown → safe HTML for kindly-note. renderMarkdown(src, opts) composes @kindly-note/lang-markdown (CommonMark tokeniser) + @kindly-note/emitters-markdown (semantic-HTML emitter with security-first defaults) + a sensible Highlighter, with option

25Downloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.1.0 (the latest release), from the OSV.dev database.

html-tokeniser

v0.0.2 · 3 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

HTML tokeniser that supports streaming interfaces. Converts HTML into tokens and provides options to transforma and convert those tokens back into a HTML string.

3Downloads across all versions in the last 7 days, from the official npm downloads API.ISCLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.0.2 (the latest release), from the OSV.dev database.

tiktoken-nodeless

v0.0.8 · 3 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

tiktoken is a fast [BPE](https://en.wikipedia.org/wiki/Byte_pair_encoding) tokeniser for use with OpenAI's models.

2Downloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.0.8 (the latest release), from the OSV.dev database.

@txm/tokeniser

v1.1.0 · 10 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

search and replace tokens from a source file, writing to a new file

4Downloads across all versions in the last 7 days, from the official npm downloads API.ISCLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.1.0 (the latest release), from the OSV.dev database.

js-tiktoken

v1.0.21 · 1 year ago

95
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

JavaScript port of tiktoken

5.7MDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.0.21 (the latest release), from the OSV.dev database.

@mrlowlevel/tiktoken-node

v0.0.8 · 3 years ago

85
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

tiktoken is a fast [BPE](https://en.wikipedia.org/wiki/Byte_pair_encoding) tokeniser for use with OpenAI's models.

1Downloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v0.0.8 (the latest release), from the OSV.dev database.

@nodable/sax

v1.2.1 · 2 months ago

100
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

Fast, Safe, Ready to Use SAX supporting Stream, incomplete input, early exit and more

3.7KDownloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.2.1 (the latest release), from the OSV.dev database.

tokeniser-package

v1.0.3 · 1 year ago

95
0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.

A simple tokenizer for encoding/decoding text into numeric tokens.

0Downloads across all versions in the last 7 days, from the official npm downloads API.MITLicense declared in the package manifest.

No known vulnerabilities

Known vulnerabilities affecting v1.0.3 (the latest release), from the OSV.dev database.
1 / 4Next