Skip to content

fix: measure normalizer scaling, not the machine it runs on - #8

Open
breken-ai wants to merge 2 commits into
v8from
harness/bf-http-2d46b575-c092-4-91fb08/bug-fix
Open

fix: measure normalizer scaling, not the machine it runs on#8
breken-ai wants to merge 2 commits into
v8from
harness/bf-http-2d46b575-c092-4-91fb08/bug-fix

Conversation

@breken-ai

Copy link
Copy Markdown
Collaborator

Problem

tests/test_ts_import_type_arguments.py::test_ts_normalizer_scales_linearly_on_large_files
passes when run alone and fails at roughly 4.1x inside the full suite. It is the only BLOCKING
failure in this repository's gate, so while it fails nothing can be published at all.

Root cause

The benchmark measures the machine's memory pressure and reports it as super-linear scaling.

Whole-suite collection supplies a mature resident heap before the benchmark runs. The old
measurement wrapped the entire call in process CPU timing, so generation-2 collector scans of
unrelated resident objects were charged to the sample. Taking independent minima then selected a
small sample with no generation-2 scan against a large sample with one or two. The ratio that
falls out is labelled super-linear by the assertion.

Measured on an unrelated machine: 2.35x in a clean process, and 2.97x once the suite's modules
are merely IMPORTED without running any of them. The small measurement gets faster and the large
one slower as resident memory grows — a bigger working set against a bigger resident set, not a
change in the algorithm's complexity. The assertion's own comment calls 3x generous; it leaves
about 27% of headroom and loading the suite consumes about 26% of it before a single test runs.

Approach

Production normalization is untouched and the 3x boundary is unchanged. Only the measurement moves:

  • Cyclic GC is disabled across each timed call and the prior state restored, so collector work is
    not attributed to the algorithm.
  • Five DISTINCT fixed-width sources at n, 2n and 4n, so no cache can hide cold work behind
    repeated identical inputs.
  • A call-import sentinel is validated after the timer stops, so the workload proves normalization
    actually happened rather than passing over a no-op.
  • MARGINAL increments are compared instead of totals, so a fixed per-call environmental cost
    cancels instead of being attributed to the algorithm.
  • The whole thing runs in a child process under a 60-second timeout whose TimeoutExpired is an
    explicit regression failure rather than a hang.

Verification

Gate at this exact head, run on an idle machine:

=== gate on harness/bf-http-2d46b575-c092-4-91fb08/bug-fix @ e49331b1 ===
PASS  skillgen-check
PASS  skillgen-audit-coverage
PASS  skillgen-schema-singleton
PASS  skillgen-monolith-rt
PASS  skillgen-always-on-rt
PASS  pytest
FAIL  bandit (exit 1)
FAIL  pip-audit (exit 1)
=== GATE: 6/8 passed ===

Every BLOCKING command passes. pytest: 5369 passed, 94 skipped, 0 failed.

The two failures are declared NON-blocking in this gate and they fail on a clean checkout too,
so they are named rather than hidden:
bandit reports 111 high-confidence findings across the
repository, and pip-audit reports PYSEC advisories for pip (PYSEC-2026-196, PYSEC-2026-3721)
and setuptools (PYSEC-2026-3447). Neither is introduced by this branch.

The correction is load-bearing, and the boundary still catches real quadratic work:

  • Removing the benchmark correction and keeping the new regression turns it red:
    scaling looks super-linear: 0.0426s -> 0.1455s (3.4x for 2x input).
  • Introducing the exact historical full-range scan turns it red at the unchanged 3x boundary:
    scaling increments look quadratic: 1.0048s -> 4.0586s (4.0x for doubled input), then reverted.
  • The new regression fails in 5 of 5 fresh no-fix processes and passes at this head in 20.16s.

Provenance, stated plainly

The work was produced by harness run bf-http-2d46b575-c092-4-91fb08 and committed there. That
run declined to publish, correctly, because the gate receipt it took said blockingPassed=false.
Every one of that receipt's eight pytest failures was a network test in tests/test_security.py,
and pip-audit in the same receipt reported Failed to resolve 'pypi.org' — the machine had no
DNS for that window. Re-running the identical gate at this identical sha on an idle machine gives
the result above. The branch was always publishable; the receipt was stale.

Impact

Two files, +105/-28, both under tests/. No production code, no public API and no output format
changes. Roll back with git revert e49331b1; the revert is test-only.

Not done

  • Python 3.10 and 3.12 timing is unmeasured — neither interpreter is available here.
  • TSX and Vue TSX scaling is unmeasured; this reproduces the TypeScript benchmark defect only.
  • Representative long-lived-process GC latency in production is unmeasured.

Problem
The blocking scaling node fails after whole-suite collection even though the indexed normalizer remains linear. Supported Python 3.11 users and the local publication gate see the false performance verdict.

Root cause
At base tests/test_ts_import_type_arguments.py:317-319, process CPU time surrounds an allocation-heavy normalizer call while cyclic GC remains enabled. Independent small and large minima select different generation-2 regimes, and the total-time ratio labels the environmental cost super-linear.

Approach
Keep production code and the 3x boundary unchanged. Disable cyclic GC only during each timed call and restore prior state. Measure distinct fixed-width cold inputs at n, 2n, and 4n; compare marginal increments; validate a normalized call-import sentinel after timing. A bounded child recollects the suite and runs the benchmark five times.

Rejected alternatives include raising the threshold, changing the indexed lookup, reusing identical inputs, trusting a discarded None result, and retaining ten outer repetitions with inadequate timeout headroom.

Verification
RED
UV_CACHE_DIR=/tmp/graphify-uv-cache UV_PROJECT_ENVIRONMENT=/home/user/graphify-lab/repo/.venv VIRTUAL_ENV=/home/user/graphify-lab/repo/.venv UV_NO_SYNC=1 uv run --frozen --active pytest tests/test_ts_normalizer_scaling_measurement.py::test_ts_normalizer_scaling_is_stable_after_suite_collection -q
F                                                                        [100%]
AssertionError: scaling looks super-linear: 0.0410s -> 0.1394s (3.4x for 2x input)
1 failed in 10.07s

Five fresh no-fix processes each failed through the original scaling assertion.

GREEN
UV_CACHE_DIR=/tmp/graphify-uv-cache UV_PROJECT_ENVIRONMENT=/home/user/graphify-lab/repo/.venv VIRTUAL_ENV=/home/user/graphify-lab/repo/.venv UV_NO_SYNC=1 uv run --frozen --active pytest tests/test_ts_normalizer_scaling_measurement.py::test_ts_normalizer_scaling_is_stable_after_suite_collection -q
.                                                                        [100%]
1 passed, 1 warning in 20.16s

MUTATION
With the benchmark fix removed and the new regression retained:
AssertionError: scaling looks super-linear: 0.0426s -> 0.1455s (3.4x for 2x input)
1 failed, 1 warning in 5.82s

HISTORICAL QUADRATIC MUTATION
AssertionError: scaling increments look quadratic: 1.0048s -> 4.0586s (4.0x for doubled input)
1 failed, 1 warning in 41.45s

The repository gate runs after this commit so its receipt can certify the exact commit SHA.

Impact
Only tests change. Public API, production behavior, and output formats are unchanged. The suite-context regression adds about 20 seconds on this machine.

Risk / rollback
Timing remains unmeasured on Python 3.10 and 3.12, TSX-specific scaling is not guarded, and production long-lived-process GC latency is outside this patch. Roll back with git revert HEAD.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants