tokenizers

Apache-2.0License declared in the package manifest.

Provides an implementation of today's most used tokenizers, with a focus on performances and versatility.

87

Security score

0 known advisories in v0.23.2

OpenSSF Scorecard 5.8/10

0-100 score computed by this site from verifiable signals only: known OSV.dev advisories affecting this exact version, how recently it was published, and the repository's OpenSSF Scorecard.advisory penalty −0 · staleness penalty −0 · scorecard component 58/100 (30% weight)

Weekly downloads

Downloads across all versions in the last 7 days, from the official npm downloads API (api.npmjs.org).

—

Unpacked size

Size of v0.23.2 on disk after npm install extracts the tarball, as reported by the npm registry.

63.6 MB

Dependencies

Direct runtime dependencies declared by v0.23.2. Transitive dependencies are not counted here — use the lockfile audit for the full tree.

0

Last publish

When v0.23.2 was published to the registry. Long gaps can indicate an unmaintained package.

14 days ago

Security advisories

Vulnerabilities affecting v0.23.2 specifically, from OSV.dev — the open database aggregating the GitHub Advisory Database (GHSA) and CVEs. Links open the full advisory.

No known vulnerabilities affect v0.23.2.

OpenSSF ScorecardAug 24, 2026

OpenSSF Scorecard is an automated audit of the source repository's security practices, run by the Open Source Security Foundation. Each check scores 0-10; hover a check for what it means. Data via deps.dev.
  • MaintainedThe project shows recent activity: commits or issue triage within the last 90 days.Result: 21 commit(s) and 4 issue activity found in the last 90 days -- score normalized to 1010
  • Code-ReviewChanges are reviewed by another maintainer before landing on the main branch.Result: Found 14/27 approved changesets -- score normalized to 55
  • CII-Best-PracticesThe project holds an OpenSSF Best Practices badge, a self-certification of security practices.Result: no effort to earn an OpenSSF best practices badge detected0
  • Binary-ArtifactsNo compiled binaries are committed to the repository — binaries can't be reviewed and may hide malicious code.Result: no binaries found in the repo10
  • Dangerous-WorkflowGitHub Actions workflows avoid dangerous patterns such as untrusted code checkout or script injection.Result: no dangerous workflow patterns detected10
  • Security-PolicyA SECURITY.md explains how to report vulnerabilities privately.Result: security policy file not detected0
  • LicenseThe project publishes a license file.Result: license file detected10
  • Token-PermissionsCI workflow tokens follow least privilege — read-only unless a job needs more.Result: detected GitHub workflow tokens with excessive permissions0
  • FuzzingThe project is fuzz-tested (e.g. OSS-Fuzz), which finds crashes and memory bugs automatically.Result: project is not fuzzed0
  • Pinned-DependenciesBuild dependencies are pinned to exact versions or hashes, preventing silent supply-chain swaps.Result: dependency not pinned by hash detected -- score normalized to 99
  • PackagingReleases are published through an automated CI/CD pipeline rather than by hand.Result: packaging workflow detected10
  • SASTStatic analysis (e.g. CodeQL) runs on pull requests to catch bugs before merge.Result: SAST tool is not run on all commits -- score normalized to 00