Versions
v0.20.1-r1 :: 0 :: gentoo
- Modified
- License
- Apache-2.0 Apache-2.0 Apache-2.0-with-LLVM-exceptions BSD-2 BSD ISC MIT MPL-2.0 Unicode-DFS-2016
- Keywords
- ~amd64
- USE flags
- debug test
USE flags
General
- debug
- Enable extra debug codepaths, like asserts and extra output. If you want to get meaningful backtraces see https://wiki.gentoo.org/wiki/Project:Quality_Assurance/Backtraces
- test
- Enable dependencies and/or preparations necessary to run tests (usually controlled by FEATURES=test but can be toggled independently)
python_single_target
- python3_10
- Build for Python 3.10 only
- python3_11
- Build for Python 3.11 only
- python3_12
- Build for Python 3.12 only
Dependencies
dev-lang / python : Freethreading (no-GIL) version of Python programming language
Runtime Dependencies
dev-lang / python : Freethreading (no-GIL) version of Python programming language
Depending packages
sci-libs / transformers : State-of-the-art Machine Learning for JAX, PyTorch and TensorFlow
Bugs
- 944852
- sci-libs/tokenizers-0.20.1-r1 fails to compile: warning: onig_sys@69.8.1: st.c:56:43: error: too many arguments to function table->type->hash
Change logs
- Repository mirror & CI · gentoo
Merge updates from master - Matt Jolly · gentoo
*/*: update for slotted Rust
Tidyups: - $(cargo_crate_uris) - Rust BDEPENDS come from the eclass except in very rare circumstances (RUST_OPTIONAL=1) - RUST_M{AX,IN}_VER where required. - Suboptimal crate separator (`-` -> `@`) Signed-off-by: Matt Jolly <kangie@gentoo.org> Closes: https://github.com/gentoo/gentoo/pull/39218 Signed-off-by: Matt Jolly <kangie@gentoo.org> - Repository mirror & CI · gentoo
Merge updates from master - Alfredo Tupone · gentoo
sci-libs/tokenizers: drop other network tests
Closes: https://bugs.gentoo.org/942327 Signed-off-by: Alfredo Tupone <tupone@gentoo.org> - Repository mirror & CI · gentoo
Merge updates from master - Alfredo Tupone · gentoo
sci-libs/tokenizers: drop failing test
Closes: https://bugs.gentoo.org/941910 Signed-off-by: Alfredo Tupone <tupone@gentoo.org> - Repository mirror & CI · gentoo
Merge updates from master - Alfredo Tupone · gentoo
sci-libs/tokenizers: add 0.20.1, drop 0.20.0
Signed-off-by: Alfredo Tupone <tupone@gentoo.org> - Repository mirror & CI · gentoo
Merge updates from master - Alfredo Tupone · gentoo
sci-libs/tokenizers: drop 0.19.1
Closes: https://bugs.gentoo.org/930015 Signed-off-by: Alfredo Tupone <tupone@gentoo.org> - Repository mirror & CI · gentoo
Merge updates from master - Alfredo Tupone · gentoo
sci-libs/tokenizers: drop 0.14.1-r1, 0.15.2-r1
Signed-off-by: Alfredo Tupone <tupone@gentoo.org> - Repository mirror & CI · gentoo
Merge updates from master - Alfredo Tupone · gentoo
sci-libs/tokenizers: add 0.20.0
Signed-off-by: Alfredo Tupone <tupone@gentoo.org> - Repository mirror & CI · gentoo
Merge updates from master - Alfredo Tupone · gentoo
sci-libs/tokenizers: update SRC_URI
Signed-off-by: Alfredo Tupone <tupone@gentoo.org> - Repository mirror & CI · gentoo
Merge updates from master - Alfredo Tupone · gentoo
sci-libs/tokenizers: upstream removed cli binary
Closes: https://bugs.gentoo.org/940780 Signed-off-by: Alfredo Tupone <tupone@gentoo.org> - Repository mirror & CI · gentoo
Merge updates from master - Alfredo Tupone · gentoo
sci-libs/tokenizers: add 0.19.1
Signed-off-by: Alfredo Tupone <tupone@gentoo.org> - Repository mirror & CI · gentoo
Merge updates from master - Alfredo Tupone · gentoo
sci-libs/tokenizers: enable test
Signed-off-by: Alfredo Tupone <tupone@gentoo.org> - Repository mirror & CI · gentoo
Merge updates from master - Alfredo Tupone · gentoo
sci-libs/tokenizers: add 0.15.2
Signed-off-by: Alfredo Tupone <tupone@gentoo.org> - Repository mirror & CI · gentoo
Merge updates from master - Alfredo Tupone · gentoo
sci-libs/tokenizers: drop QA_CHECK for musl build, too
Closes: https://bugs.gentoo.org/924970 Signed-off-by: Alfredo Tupone <tupone@gentoo.org> - Repository mirror & CI · gentoo
Merge updates from master - Alfredo Tupone · gentoo
sci-libs/tokenizers: PythonCompatUpdate
Signed-off-by: Alfredo Tupone <tupone@gentoo.org> - Repository mirror & CI · gentoo
Merge updates from master - Alfredo Tupone · gentoo
sci-libs/tokenizers: drop 0.13.3
Signed-off-by: Alfredo Tupone <tupone@gentoo.org> - Repository mirror & CI · gentoo
Merge updates from master - Alfredo Tupone · gentoo
sci-libs/tokenizers: add QA_FLAGS_IGNORED. It's rust
Closes: https://bugs.gentoo.org/904231 Signed-off-by: Alfredo Tupone <tupone@gentoo.org> - Repository mirror & CI · gentoo
Merge updates from master - Alfredo Tupone · gentoo
sci-libs/tokenizers: add 0.14.1, drop 0.14.0
Signed-off-by: Alfredo Tupone <tupone@gentoo.org> - Repository mirror & CI · gentoo
Merge updates from master - Alfredo Tupone · gentoo
sci-libs/tokenizers: add 0.14.0
Signed-off-by: Alfredo Tupone <tupone@gentoo.org> - Repository mirror & CI · gentoo
Merge updates from master - Alfredo Tupone · gentoo
sci-libs/tokenizers: add dep on setuptools_rust
Closes: https://bugs.gentoo.org/904216 Signed-off-by: Alfredo Tupone <tupone@gentoo.org> - Repository mirror & CI · gentoo
Merge updates from master - Alfredo Tupone · gentoo
sci-libs/tokenizers: new package, add 0.13.3
Signed-off-by: Alfredo Tupone <tupone@gentoo.org>