sci-libs / tokenizers

Implementation of today's most used tokenizers

Official package sites : https://github.com/huggingface/tokenizers ·

v0.20.1-r1 :: 0 :: gentoo

Modified
License
Apache-2.0 Apache-2.0 Apache-2.0-with-LLVM-exceptions BSD-2 BSD ISC MIT MPL-2.0 Unicode-DFS-2016
Keywords
~amd64
USE flags
debug test

General

debug
Enable extra debug codepaths, like asserts and extra output. If you want to get meaningful backtraces see https://wiki.gentoo.org/wiki/Project:Quality_Assurance/Backtraces
test
Enable dependencies and/or preparations necessary to run tests (usually controlled by FEATURES=test but can be toggled independently)

python_single_target

python3_10
Build for Python 3.10 only
python3_11
Build for Python 3.11 only
python3_12
Build for Python 3.12 only

dev-lang / python : Freethreading (no-GIL) version of Python programming language

dev-lang / python : Freethreading (no-GIL) version of Python programming language

sci-libs / transformers : State-of-the-art Machine Learning for JAX, PyTorch and TensorFlow

944852
sci-libs/tokenizers-0.20.1-r1 fails to compile: warning: onig_sys@69.8.1: st.c:56:43: error: too many arguments to function table->type->hash
Repository mirror & CI · gentoo
Merge updates from master
Matt Jolly · gentoo
*/*: update for slotted Rust
Tidyups: - $(cargo_crate_uris) - Rust BDEPENDS come from the eclass except in very rare circumstances (RUST_OPTIONAL=1) - RUST_M{AX,IN}_VER where required. - Suboptimal crate separator (`-` -> `@`) Signed-off-by: Matt Jolly <kangie@gentoo.org> Closes: https://github.com/gentoo/gentoo/pull/39218 Signed-off-by: Matt Jolly <kangie@gentoo.org>
Repository mirror & CI · gentoo
Merge updates from master
Alfredo Tupone · gentoo
sci-libs/tokenizers: drop other network tests
Closes: https://bugs.gentoo.org/942327 Signed-off-by: Alfredo Tupone <tupone@gentoo.org>
Repository mirror & CI · gentoo
Merge updates from master
Alfredo Tupone · gentoo
sci-libs/tokenizers: drop failing test
Closes: https://bugs.gentoo.org/941910 Signed-off-by: Alfredo Tupone <tupone@gentoo.org>
Repository mirror & CI · gentoo
Merge updates from master
Alfredo Tupone · gentoo
sci-libs/tokenizers: add 0.20.1, drop 0.20.0
Signed-off-by: Alfredo Tupone <tupone@gentoo.org>
Repository mirror & CI · gentoo
Merge updates from master
Alfredo Tupone · gentoo
sci-libs/tokenizers: drop 0.19.1
Closes: https://bugs.gentoo.org/930015 Signed-off-by: Alfredo Tupone <tupone@gentoo.org>
Repository mirror & CI · gentoo
Merge updates from master
Alfredo Tupone · gentoo
sci-libs/tokenizers: drop 0.14.1-r1, 0.15.2-r1
Signed-off-by: Alfredo Tupone <tupone@gentoo.org>
Repository mirror & CI · gentoo
Merge updates from master
Alfredo Tupone · gentoo
sci-libs/tokenizers: add 0.20.0
Signed-off-by: Alfredo Tupone <tupone@gentoo.org>
Repository mirror & CI · gentoo
Merge updates from master
Alfredo Tupone · gentoo
sci-libs/tokenizers: update SRC_URI
Signed-off-by: Alfredo Tupone <tupone@gentoo.org>
Repository mirror & CI · gentoo
Merge updates from master
Alfredo Tupone · gentoo
sci-libs/tokenizers: upstream removed cli binary
Closes: https://bugs.gentoo.org/940780 Signed-off-by: Alfredo Tupone <tupone@gentoo.org>
Repository mirror & CI · gentoo
Merge updates from master
Alfredo Tupone · gentoo
sci-libs/tokenizers: add 0.19.1
Signed-off-by: Alfredo Tupone <tupone@gentoo.org>
Repository mirror & CI · gentoo
Merge updates from master
Alfredo Tupone · gentoo
sci-libs/tokenizers: enable test
Signed-off-by: Alfredo Tupone <tupone@gentoo.org>
Repository mirror & CI · gentoo
Merge updates from master
Alfredo Tupone · gentoo
sci-libs/tokenizers: add 0.15.2
Signed-off-by: Alfredo Tupone <tupone@gentoo.org>
Repository mirror & CI · gentoo
Merge updates from master
Alfredo Tupone · gentoo
sci-libs/tokenizers: drop QA_CHECK for musl build, too
Closes: https://bugs.gentoo.org/924970 Signed-off-by: Alfredo Tupone <tupone@gentoo.org>
Repository mirror & CI · gentoo
Merge updates from master
Alfredo Tupone · gentoo
sci-libs/tokenizers: PythonCompatUpdate
Signed-off-by: Alfredo Tupone <tupone@gentoo.org>
Repository mirror & CI · gentoo
Merge updates from master
Alfredo Tupone · gentoo
sci-libs/tokenizers: drop 0.13.3
Signed-off-by: Alfredo Tupone <tupone@gentoo.org>
Repository mirror & CI · gentoo
Merge updates from master
Alfredo Tupone · gentoo
sci-libs/tokenizers: add QA_FLAGS_IGNORED. It's rust
Closes: https://bugs.gentoo.org/904231 Signed-off-by: Alfredo Tupone <tupone@gentoo.org>
Repository mirror & CI · gentoo
Merge updates from master
Alfredo Tupone · gentoo
sci-libs/tokenizers: add 0.14.1, drop 0.14.0
Signed-off-by: Alfredo Tupone <tupone@gentoo.org>
Repository mirror & CI · gentoo
Merge updates from master
Alfredo Tupone · gentoo
sci-libs/tokenizers: add 0.14.0
Signed-off-by: Alfredo Tupone <tupone@gentoo.org>
Repository mirror & CI · gentoo
Merge updates from master
Alfredo Tupone · gentoo
sci-libs/tokenizers: add dep on setuptools_rust
Closes: https://bugs.gentoo.org/904216 Signed-off-by: Alfredo Tupone <tupone@gentoo.org>
Repository mirror & CI · gentoo
Merge updates from master
Alfredo Tupone · gentoo
sci-libs/tokenizers: new package, add 0.13.3
Signed-off-by: Alfredo Tupone <tupone@gentoo.org>