fix: relax tokenizers version constraint

[nightcityblade]

Summary

Relax the tokenizers dependency from a single pinned version to a bounded range that supports both the current Transformers 4.46 stack and newer Transformers releases through 4.57.

Fixes #508

List of files changed and why

• setup.py - allow tokenizers>=0.20.3,<0.23 instead of only 0.20.3.
• requirements/server.txt - keep server requirements aligned with package metadata.

How Has This Been Tested?

• python3 -m py_compile setup.py
• /tmp/whisperlive-venv/bin/python -m pip install --dry-run 'transformers==4.57.1' 'tokenizers>=0.20.3,<0.23'\n- /tmp/whisperlive-venv/bin/python -m pip install --dry-run 'transformers==4.46.3' 'tokenizers>=0.20.3,<0.23'\n

[boxerab]

Hello @nightcityblade, thanks for the PR. Can you please take a look at failing CI ? 🙏

[nightcityblade]

I checked the failed run in detail. The breakage is in the server inference tests, where / were never created before the assertions ran. That doesn't appear to come from the tokenizers pin change itself, so this looks more like a test/server-startup failure than a dependency-resolution failure. I can't rerun the workflow from my side because GitHub reports I don't have the required repository permissions, but I'm happy to dig further if the next run shows the same pattern.

[Copilot]

Pull request overview

This PR relaxes the tokenizers dependency constraint to allow compatibility with both the existing Transformers 4.46.x stack and newer Transformers releases (through 4.57.x), addressing dependency resolution conflicts reported in #508.

Changes:

• Update setup.py to allow tokenizers>=0.20.3,<0.23 instead of hard-pinning tokenizers==0.20.3.
• Update requirements/server.txt to match the same tokenizers constraint for server installs.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
setup.py	Relaxes package install dependency to a bounded tokenizers range to support multiple Transformers versions.
requirements/server.txt	Keeps server requirements aligned with the package’s tokenizers constraint.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[boxerab]

I checked the failed run in detail. The breakage is in the server inference tests, where / were never created before the assertions ran. That doesn't appear to come from the tokenizers pin change itself, so this looks more like a test/server-startup failure than a dependency-resolution failure. I can't rerun the workflow from my side because GitHub reports I don't have the required repository permissions, but I'm happy to dig further if the next run shows the same pattern.

Thanks, I did run CI a couple of times on the PR and it always fails, so probably not due to flaky tests. If you could dig deeper that would be very much appreciated. 🙏