sci-libs / tokenizers

Implementation of today's most used tokenizers

Official package sites : https://github.com/huggingface/tokenizers ·

v0.15.2-r1 :: 0 :: gentoo

Modified
License
Apache-2.0 Apache-2.0 Apache-2.0-with-LLVM-exceptions BSD-2 BSD ISC MIT MPL-2.0 Unicode-DFS-2016
Keywords
~amd64
USE flags
debug test

v0.14.1-r1 :: 0 :: gentoo

Modified
License
Apache-2.0 Apache-2.0 Apache-2.0-with-LLVM-exceptions BSD-2 BSD ISC MIT MPL-2.0 Unicode-DFS-2016
Keywords
~amd64
USE flags
debug test

General

debug
Enable extra debug codepaths, like asserts and extra output. If you want to get meaningful backtraces see https://wiki.gentoo.org/wiki/Project:Quality_Assurance/Backtraces
test
Enable dependencies and/or preparations necessary to run tests (usually controlled by FEATURES=test but can be toggled independently)

python_single_target

python3_10
Build for Python 3.10 only
python3_11
Build for Python 3.11 only
python3_12
Build for Python 3.12 only

python_targets

python3_10
Build with Python 3.10
python3_11
Build with Python 3.11
python3_12
Build with Python 3.12

dev-lang / python : An interpreted, interactive, object-oriented programming language

dev-lang / python : An interpreted, interactive, object-oriented programming language

sci-libs / transformers : State-of-the-art Machine Learning for JAX, PyTorch and TensorFlow

930015
sci-libs/tokenizers-0.15.2-r1 fails tests: FAILED test_tokenizer.py::TestTokenizer::test_encode_formats - TypeError: sep_token not found in the vocabulary
Repository mirror & CI · gentoo
Merge updates from master
Alfredo Tupone · gentoo
sci-libs/tokenizers: enable test
Signed-off-by: Alfredo Tupone <tupone@gentoo.org>
Repository mirror & CI · gentoo
Merge updates from master
Alfredo Tupone · gentoo
sci-libs/tokenizers: add 0.15.2
Signed-off-by: Alfredo Tupone <tupone@gentoo.org>
Repository mirror & CI · gentoo
Merge updates from master
Alfredo Tupone · gentoo
sci-libs/tokenizers: drop QA_CHECK for musl build, too
Closes: https://bugs.gentoo.org/924970 Signed-off-by: Alfredo Tupone <tupone@gentoo.org>
Repository mirror & CI · gentoo
Merge updates from master
Alfredo Tupone · gentoo
sci-libs/tokenizers: PythonCompatUpdate
Signed-off-by: Alfredo Tupone <tupone@gentoo.org>
Repository mirror & CI · gentoo
Merge updates from master
Alfredo Tupone · gentoo
sci-libs/tokenizers: drop 0.13.3
Signed-off-by: Alfredo Tupone <tupone@gentoo.org>
Repository mirror & CI · gentoo
Merge updates from master
Alfredo Tupone · gentoo
sci-libs/tokenizers: add QA_FLAGS_IGNORED. It's rust
Closes: https://bugs.gentoo.org/904231 Signed-off-by: Alfredo Tupone <tupone@gentoo.org>
Repository mirror & CI · gentoo
Merge updates from master
Alfredo Tupone · gentoo
sci-libs/tokenizers: add 0.14.1, drop 0.14.0
Signed-off-by: Alfredo Tupone <tupone@gentoo.org>
Repository mirror & CI · gentoo
Merge updates from master
Alfredo Tupone · gentoo
sci-libs/tokenizers: add 0.14.0
Signed-off-by: Alfredo Tupone <tupone@gentoo.org>
Repository mirror & CI · gentoo
Merge updates from master
Alfredo Tupone · gentoo
sci-libs/tokenizers: add dep on setuptools_rust
Closes: https://bugs.gentoo.org/904216 Signed-off-by: Alfredo Tupone <tupone@gentoo.org>
Repository mirror & CI · gentoo
Merge updates from master
Alfredo Tupone · gentoo
sci-libs/tokenizers: new package, add 0.13.3
Signed-off-by: Alfredo Tupone <tupone@gentoo.org>