tokenizers: Fast State-of-the-Art Tokenizers optimized for Research and Production1

Provides an implementation of today's most used tokenizers, with a focus on performance and versatility.

... part of T2, get it here

URL: https://github.com/huggingface/tokenizers

Author: Anthony MOI <m [dot] anthony [dot] moi [at] gmail [dot] com>
Maintainer: The T2 Project <t2 [at] t2-project [dot] org>

License: APL
Status: Stable
Version: 0.22.0

Download: https://github.com/huggingface/tokenizers/ tokenizers-0.22.0.tar.gz

T2 source: tokenizers.cache
T2 source: tokenizers.desc

Build time (on reference hardware): 120% (relative to binutils)2

Installed size (on reference hardware): 9.66 MB, 43 files

Dependencies (build time detected): bash coreutils curl diffutils gawk grep gzip linux-header openssl pyrequests python python-gpep517 python-maturin rustc sed tar

Installed files (on reference hardware): [show]

1) This page was automatically generated from the T2 package source. Corrections, such as dead links, URL changes or typos need to be performed directly on that source.

2) Compatible with Linux From Scratch's "Standard Build Unit" (SBU).

Generated: in 0.07s — Copyright ©2004-2025 ExactCODE GmbH and others. All Rights Reserved. The T2 SDE Project is maintained
and driven by the community and sponsored by ExactCODE. This is a community maintained site. ExactCODE is not responsible for the content.