Repository navigation
spaCy 3.8.16 tokenizer OverflowError on Windows after upgrading preshed 3.0.13 → 3.0.14 #14058
Description
Title: spaCy 3.8.16 tokenizer OverflowError on Windows after upgrading preshed 3.0.13 → 3.0.14
Description
Loading en_core_web_sm 3.8.0 fails with an OverflowError during tokenizer deserialization when using preshed==3.0.14. Reverting to preshed==3.0.13 resolves the failure.
Minimal reproduction
Reproduced in a clean Windows Python 3.12 virtual environment using pip, independently of any application or project-specific dependencies.
py -3.12 -m venv .venv
.\.venv\Scripts\python.exe -m pip install "spacy==3.8.16" "preshed==3.0.13" "en-core-web-sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl"
.\.venv\Scripts\python.exe -c "import spacy; print(spacy.load('en_core_web_sm').pipe_names)"Result: Success.
Then upgrade only Preshed:
.\.venv\Scripts\python.exe -m pip install --no-deps "preshed==3.0.14"
.\.venv\Scripts\python.exe -c "import spacy; print(spacy.load('en_core_web_sm').pipe_names)"Result: Failure.
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\<user>\local_projects\preshed-repro\.venv\Lib\site-packages\spacy\__init__.py", line 74, in load
return util.load_model(
^^^^^^^^^^^^^^^^
File "C:\Users\<user>\local_projects\preshed-repro\.venv\Lib\site-packages\spacy\util.py", line 524, in load_model
return load_model_from_package(name, **kwargs) # type: ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\<user>\local_projects\preshed-repro\.venv\Lib\site-packages\spacy\util.py", line 560, in load_model_from_package
return cls.load(
^^^^^^^^^
File "C:\Users\<user>\local_projects\preshed-repro\.venv\Lib\site-packages\en_core_web_sm\__init__.py", line 10, in load
return load_model_from_init_py(__file__, **overrides)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\<user>\local_projects\preshed-repro\.venv\Lib\site-packages\spacy\util.py", line 743, in load_model_from_init_py
return load_model_from_path(
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\<user>\local_projects\preshed-repro\.venv\Lib\site-packages\spacy\util.py", line 608, in load_model_from_path
return nlp.from_disk(model_path, exclude=exclude, overrides=overrides)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\<user>\local_projects\preshed-repro\.venv\Lib\site-packages\spacy\language.py", line 2249, in from_disk
util.from_disk(path, deserializers, exclude) # type: ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\<user>\local_projects\preshed-repro\.venv\Lib\site-packages\spacy\util.py", line 1450, in from_disk
reader(path / key)
File "C:\Users\<user>\local_projects\preshed-repro\.venv\Lib\site-packages\spacy\language.py", line 2235, in <lambda>
deserializers["tokenizer"] = lambda p: self.tokenizer.from_disk( # type: ignore[union-attr]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "spacy/tokenizer.pyx", line 787, in spacy.tokenizer.Tokenizer.from_disk
self.from_bytes(bytes_data, exclude=exclude)
File "spacy/tokenizer.pyx", line 833, in spacy.tokenizer.Tokenizer.from_bytes
self.rules = {}
File "spacy/tokenizer.pyx", line 128, in spacy.tokenizer.Tokenizer.rules.__set__
self._flush_specials()
File "spacy/tokenizer.pyx", line 239, in spacy.tokenizer.Tokenizer._flush_specials
del self._specials[k]
OverflowError: Python int too large to convert to C ssize_t
The same failure was initially observed in a Python 3.11 environment and subsequently reproduced independently with Python 3.12.
Info about spaCy
- spaCy version: 3.8.16
- Platform: Windows-11-10.0.26200-SP0
- Python version: 3.12.6
- Pipelines: en_core_web_sm (3.8.0)
Activity
boxydog commentedon Oct 25, 2026
Same failure here, and it isn't Windows-specific. spaCy 3.8.16, en_core_web_sm 3.8.0:
Same traceback, ending in
Tokenizer._flush_specials->del self._specials[k]->OverflowError: Python int too large to convert to C ssize_t.On macOS, going back to preshed 3.0.13 fixes it, with cymem 2.0.16, murmurhash 1.0.16, numpy 2.5.4 and srsly 2.5.5 left at their new versions. Pinning
preshed<3.0.14works as a stopgap.fix(pii): the install hint names the qwed[pii] extra
fix(deps): exclude preshed 3.0.14 from the pii extra (#516)
ci(release): test with the latest dependencies before publishing
fix(analyzer): pin preshed<3.0.14 to keep spaCy model loading working (…
SergeyShk commentedon Oct 25, 2026
Same here with spaCy 3.8.15 as well as 3.8.16 (macOS arm64, CPython 3.11-3.14). A repro without a pipeline package:
It passes with
preshed==3.0.13, and fails the same way withspacy==3.8.15. Except on Python 3.13, it reproduces without spaCy. With 3.0.14, setting, getting and deleting any key from 2**63 raise the same error, while 3.0.13 accepts them:These direct calls fail on Python 3.11, 3.12 and 3.14. On 3.13 they pass, and only the tokenizer fails.
The cause looks like the Cython used to build the wheels, not a change in preshed:
python_requires(<3.16) and the Python 3.15 classifier.maps.*.soof 3.0.13 contains_cython_3_2_4, and that of 3.0.14 contains_cython_3_3_0. The build requirement iscython>=3.1, unpinned.cython==3.2.4as a build constraint makes the repro pass. Building it withcython==3.3.0makes it fail, and so does building 3.0.13 with it:So any build of preshed from source now gets Cython 3.3.0 and breaks, on a platform without wheels or with
--no-binary. Pinning Cython below 3.3 in the build requirements and re-releasing (e.g. as 3.0.15) would likely fix it. Meanwhilepreshed!=3.0.14(with the wheels) works as a stopgapfix(deps): pin preshed<3.0.14, which breaks spaCy model loading
[FIX] HORILLA: Pin preshed to 3.0.13, 3.0.14 breaks spaCy model loading
chore(release): 0.6.2
acoustesh commentedon Oct 25, 2026
Confirmed that
preshed==3.0.15fixes this. On Linux, Python 3.14.8, spaCy 3.8.16 withes_core_news_sm3.8.0,spacy.load()raised the sameOverflowErrorinTokenizer._flush_specialswith preshed 3.0.14. It loads fine with 3.0.13 and with 3.0.15, so thepreshed<3.0.14pin should no longer be needed once 3.0.15 is resolved.ldej commentedon Oct 29, 2026
Same failure on a different interpreter and platform, which supports preshed being the cause rather than spaCy or the OS.
Linux x86_64, CPython 3.15.2, spaCy 3.8.18 from the published cp315 wheel,
en_core_web_sm3.8.0:spacy.load("en_core_web_sm")OverflowError: Python int too large to convert to C ssize_tinTokenizer._flush_specialsOnly preshed changed between the two runs, so the tokenizer crash tracks preshed rather than the interpreter, and 3.0.15 resolves it here too.
One downstream consequence if it helps triage:
presidio-analyzer2.2.366 pinspreshed<3.0.14, so anyone using it on 3.15 gets the broken build unless they override the pin. I opened data-privacy-stack/presidio#2431 about relaxing it.fix(analyzer): exclude only broken preshed 3.0.14 instead of capping …
fix(deps): move presidio-analyzer to 2.2.366 for transformers 5 and b…