bpe-tokenizer
BSD-2-ClauseBuild your own vocabulary from application-specific corpus using Byte pair encoding (BPE) algorithm.
90
Security score
0 known advisories in v2.2.0
Weekly downloads
—
Unpacked size
59.3 kB
Dependencies
0
Last publish
2 years ago
Security advisories
No known vulnerabilities affect v2.2.0.