Repository navigation
Build abi3t wheels for Python 3.15 support #2488
Description
Activity
It looks like this is currently blocked on rust-numpy, see PyO3/rust-numpy#556 (comment).
It seems everything works using rust-numpy built from source.
However, there are some slowdowns on benchmarks:
Call 3.15 abi3 3.15 abi3t 3.15t cp315t 3.15t abi3t Cause list getters ( .ids×3 +len)470 ns 517 ns (+10%) 406 ns 871 ns (+115%) free-threaded: PyList_SetItemlocks per element; GIL build: ~5 ns per pyclass method callids_array(before fix)700 ns 829 ns (+18%) 629 ns 760 ns (+21%) rust-numpy borrow tracking, 18 NumPy C-API calls per call ids_array(after tokenizers fix)320 ns 334 ns (+4%) 249 ns 255 ns (+2%) decode(ndarray)507 ns 586 ns (+16%) 465 ns 525 ns (+13%) 20 _PyArray_GET_ITEM_DATAcalls per calldecode(list)661 ns 689 ns (+4%) 632 ns 645 ns (+2%) PyIter_NextItem, ~9 instructions per elementThis was produced using AI assistance. The cause column is speculation from an AI model and may not be completely accurate.
Feature request
Maturin/PyO3 support building abi3.abi3t wheels that are importable on both free-threaded and GIL-enabled 3.15. Tokenizers can then support both builds with a single wheel and support all non-EOL python versions with three wheels per platform.
Motivation
Python 3.15 support will enable projects that depend on tokenizers to start testing on Python 3.15 and add support themselves.
Your contribution
I’m happy to submit a PR for this. I also added abi3t support to both Maturin and PyO3 so I’m in a good position to fix any issues that come up.