fix: measure normalizer scaling, not the machine it runs on - #8
Open
breken-ai wants to merge 2 commits into
Open
fix: measure normalizer scaling, not the machine it runs on#8breken-ai wants to merge 2 commits into
breken-ai wants to merge 2 commits into
Conversation
Problem The blocking scaling node fails after whole-suite collection even though the indexed normalizer remains linear. Supported Python 3.11 users and the local publication gate see the false performance verdict. Root cause At base tests/test_ts_import_type_arguments.py:317-319, process CPU time surrounds an allocation-heavy normalizer call while cyclic GC remains enabled. Independent small and large minima select different generation-2 regimes, and the total-time ratio labels the environmental cost super-linear. Approach Keep production code and the 3x boundary unchanged. Disable cyclic GC only during each timed call and restore prior state. Measure distinct fixed-width cold inputs at n, 2n, and 4n; compare marginal increments; validate a normalized call-import sentinel after timing. A bounded child recollects the suite and runs the benchmark five times. Rejected alternatives include raising the threshold, changing the indexed lookup, reusing identical inputs, trusting a discarded None result, and retaining ten outer repetitions with inadequate timeout headroom. Verification RED UV_CACHE_DIR=/tmp/graphify-uv-cache UV_PROJECT_ENVIRONMENT=/home/user/graphify-lab/repo/.venv VIRTUAL_ENV=/home/user/graphify-lab/repo/.venv UV_NO_SYNC=1 uv run --frozen --active pytest tests/test_ts_normalizer_scaling_measurement.py::test_ts_normalizer_scaling_is_stable_after_suite_collection -q F [100%] AssertionError: scaling looks super-linear: 0.0410s -> 0.1394s (3.4x for 2x input) 1 failed in 10.07s Five fresh no-fix processes each failed through the original scaling assertion. GREEN UV_CACHE_DIR=/tmp/graphify-uv-cache UV_PROJECT_ENVIRONMENT=/home/user/graphify-lab/repo/.venv VIRTUAL_ENV=/home/user/graphify-lab/repo/.venv UV_NO_SYNC=1 uv run --frozen --active pytest tests/test_ts_normalizer_scaling_measurement.py::test_ts_normalizer_scaling_is_stable_after_suite_collection -q . [100%] 1 passed, 1 warning in 20.16s MUTATION With the benchmark fix removed and the new regression retained: AssertionError: scaling looks super-linear: 0.0426s -> 0.1455s (3.4x for 2x input) 1 failed, 1 warning in 5.82s HISTORICAL QUADRATIC MUTATION AssertionError: scaling increments look quadratic: 1.0048s -> 4.0586s (4.0x for doubled input) 1 failed, 1 warning in 41.45s The repository gate runs after this commit so its receipt can certify the exact commit SHA. Impact Only tests change. Public API, production behavior, and output formats are unchanged. The suite-context regression adds about 20 seconds on this machine. Risk / rollback Timing remains unmeasured on Python 3.10 and 3.12, TSX-specific scaling is not guarded, and production long-lived-process GC latency is outside this patch. Roll back with git revert HEAD.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
tests/test_ts_import_type_arguments.py::test_ts_normalizer_scales_linearly_on_large_filespasses when run alone and fails at roughly 4.1x inside the full suite. It is the only BLOCKING
failure in this repository's gate, so while it fails nothing can be published at all.
Root cause
The benchmark measures the machine's memory pressure and reports it as super-linear scaling.
Whole-suite collection supplies a mature resident heap before the benchmark runs. The old
measurement wrapped the entire call in process CPU timing, so generation-2 collector scans of
unrelated resident objects were charged to the sample. Taking independent minima then selected a
small sample with no generation-2 scan against a large sample with one or two. The ratio that
falls out is labelled super-linear by the assertion.
Measured on an unrelated machine: 2.35x in a clean process, and 2.97x once the suite's modules
are merely IMPORTED without running any of them. The small measurement gets faster and the large
one slower as resident memory grows — a bigger working set against a bigger resident set, not a
change in the algorithm's complexity. The assertion's own comment calls 3x generous; it leaves
about 27% of headroom and loading the suite consumes about 26% of it before a single test runs.
Approach
Production normalization is untouched and the 3x boundary is unchanged. Only the measurement moves:
not attributed to the algorithm.
repeated identical inputs.
actually happened rather than passing over a no-op.
cancels instead of being attributed to the algorithm.
explicit regression failure rather than a hang.
Verification
Gate at this exact head, run on an idle machine:
Every BLOCKING command passes.
pytest: 5369 passed, 94 skipped, 0 failed.The two failures are declared NON-blocking in this gate and they fail on a clean checkout too,
so they are named rather than hidden:
banditreports 111 high-confidence findings across therepository, and
pip-auditreports PYSEC advisories forpip(PYSEC-2026-196, PYSEC-2026-3721)and
setuptools(PYSEC-2026-3447). Neither is introduced by this branch.The correction is load-bearing, and the boundary still catches real quadratic work:
scaling looks super-linear: 0.0426s -> 0.1455s (3.4x for 2x input).scaling increments look quadratic: 1.0048s -> 4.0586s (4.0x for doubled input), then reverted.Provenance, stated plainly
The work was produced by harness run
bf-http-2d46b575-c092-4-91fb08and committed there. Thatrun declined to publish, correctly, because the gate receipt it took said
blockingPassed=false.Every one of that receipt's eight pytest failures was a network test in
tests/test_security.py,and
pip-auditin the same receipt reportedFailed to resolve 'pypi.org'— the machine had noDNS for that window. Re-running the identical gate at this identical sha on an idle machine gives
the result above. The branch was always publishable; the receipt was stale.
Impact
Two files, +105/-28, both under
tests/. No production code, no public API and no output formatchanges. Roll back with
git revert e49331b1; the revert is test-only.Not done