From 84be64150e85d2268649af7137552c843c4f9c9b Mon Sep 17 00:00:00 2001 From: Samran Asif Date: Mon, 7 Sep 2026 14:39:58 +0500 Subject: [PATCH 1/2] chore(release): 0.2.0 Ten open issues closed this pass, and the theme running through them is worth naming in the changelog rather than only in the individual entries: several documented capabilities were declared and never wired up -- an entry-point group nothing read, a descriptor format nothing loaded, an inference feature with no inference -- and several rules reported the opposite of what had happened. Version synchronised across pyproject, CITATION.cff and the CHANGELOG, with a test that they agree. ToolTrace Bench shipped with those three saying three different things, two of them describing a version that had never been tagged; it is a one-line fix every release, which is exactly what a test is for. Confirmed it fails when they drift. Co-Authored-By: Claude Opus 5 --- CHANGELOG.md | 101 +++++++++++++++++++++++++++++++ CITATION.cff | 4 +- pyproject.toml | 2 +- tests/unit/test_readme_counts.py | 29 +++++++++ 4 files changed, 133 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 16d0949..8ce8e43 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,107 @@ All notable changes. Format based on Keep a Changelog; versions are semver. ## [Unreleased] +## [0.2.0] - 2026-09-07 + +Ten open issues closed. The theme running through them is that several +documented capabilities were declared and never wired up, and several rules +reported the opposite of what had happened. + +### Added — protocols + +- **AsyncAPI 3.x** (#18). The adapter handled 2.x only. Direction is now + normalized to the application's point of view, because the two versions' + words are inverted -- AsyncAPI 2's `publish` describes what the application + *consumes* and `subscribe` what it *produces*, while 3's `send`/`receive` + read from the application's side. Storing the raw word made a 2.x document + and its own 3.x migration compare as two unrelated contracts. The adapter is + also registered under the `apiverity.specs` entry point, which it was not. +- **Compiled gRPC descriptor sets** (#17): `.desc`/`.pb`/`.protoset` load, + decoded from the protobuf wire format with no protobuf runtime dependency. + For a proto with imports this is the only input that can be correct. +- **GraphQL operation testing** (#16): schema-driven query generation, + persisted operation documents via `test --operations`, `{data, errors}` + envelope assertions, and introspection-based drift via `drift --base-url`. + GraphQL cannot go through the HTTP runner -- a server answers a malformed + query with 200 and an `errors` array, so a status-code verdict marks every + failure a pass. + +### Added — analysis + +- **Corpus drift** (#21): `drift --corpus traffic.har` aggregates findings with + a frequency per operation and separates systematic drift from one-offs. A + thousand entries against a service missing one declared header used to + produce a thousand identical findings. +- **Statistically sound performance gates** (#24): bootstrap confidence + intervals for percentiles and throughput, a Wilson interval for error rate, + `--warmup`, and per-metric `--tolerance`. A change must clear the tolerance + *and* have non-overlapping intervals before it is called a regression; where + they overlap the run is reported inconclusive rather than passed. Measured + against a fixed local target over ten runs, this took asserted regressions + from 22 to 8 -- all of them false by construction, because nothing changed. +- **Pluggable case generators** (#19): the `apiverity.generators` entry point + was declared, documented and never read. Four strategies ship with it -- + unicode, nesting, numeric boundaries and defensive header safety. +- **Workflow inference** (#20): `workflow --infer` drafts a manifest from the + `links` a spec declares, and only from those. Every step is emitted + commented out; destructive steps are commented twice, so uncommenting the + file wholesale does not arm them. +- Seven protobuf compatibility rules and the rest of the request/response + requiredness catalog (#15, #17). The catalog is 44 rules, generated. + +### Fixed — rules that reported the opposite of what happened + +- A field **becoming required** in a request body was reported as + `BRK-PARAM-OPTIONALIZED` at INFO. Requiredness was only compared in one + direction and passed no new value, so a breaking change passed a CI gate as + informational. A response field becoming optional was not detected at all. +- Dropping a required field from an AsyncAPI message the application **sends** + was reported as a request relaxation -- "senders are unaffected" -- when the + application is the sender and the consumers break. +- A **unary gRPC RPC becoming bidirectional** produced zero changes. The + `stream` markers were captured by the parser and discarded. +- A protobuf field **renamed at the same number** is wire-compatible and is now + reported as such at WARN; only a number whose type changed is an error. + Reporting every rename as data corruption trains people to ignore the rule + that catches actual corruption. + +### Fixed — checks that were never reached + +- `PROTO-FIELD-NUMBER-REUSE` was constructed on every duplicate field number + and then dropped: the function returned only the schema. +- `detect_drift` accepted `forbid_undeclared_fields` and passed a hardcoded + `True`. +- `EXIT_UNREACHABLE` was unreachable. `measure` catches connection errors per + request, so a target with nothing listening produced a clean report and + exit 0. +- `import_har` called `json.loads` on every postData; one HTML error page took + down the import of an entire corpus. + +### Fixed — reports + +- **SARIF carries locations** (#22). Findings have always had file, line, + column and JSON pointer; the renderer emitted none of it, so GitHub code + scanning annotated the repository rather than the line. Adds rule metadata + and `partialFingerprints` that exclude the message, so an alert is not + retired and re-raised when a number in its text changes. +- **JUnit emits testcases.** It declared `tests="N"` over an empty testsuite, + so every consumer read zero tests. +- **HTML escapes spec content** and is fully self-contained, with filter state + in the URL hash. Messages carry field names straight out of a user's + document and were interpolated raw. +- `apiverity report` no longer carries its own copy of every renderer; the two + had already diverged. + +### Changed + +- The demo app loads page groups as separate chunks (#23): 224 kB in one file + to a 198 kB entry plus 31 kB across six lazy chunks. +- Docs corrected where they described capabilities that did not exist: + AsyncAPI was "planned" while a 2.x adapter shipped, gRPC descriptor import + was "EXISTING" with nothing in the plugin mentioning descriptors, and + workflow inference was "PARTIAL" with no inference of any kind. +- README count claims are re-derived and tested, not restated. + ### Fixed - **Console script entry point** pointed at a nonexistent symbol (`apiverity.cli.main:cli`); installing the package produced an `apiverity` diff --git a/CITATION.cff b/CITATION.cff index 62664ec..19cd28a 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -21,5 +21,5 @@ keywords: - breaking-changes - drift-detection license: Apache-2.0 -version: 0.1.0 -date-released: '2026-08-22' \ No newline at end of file +version: 0.2.0 +date-released: '2026-09-07' \ No newline at end of file diff --git a/pyproject.toml b/pyproject.toml index d0101ea..c20784a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "api-verity-lab" -version = "0.1.0" +version = "0.2.0" description = "Unified API contract governance, breaking-change analysis, schema-driven testing, runtime drift detection, traffic replay and performance regression for OpenAPI, GraphQL and gRPC." readme = "README.md" license = { text = "Apache-2.0" } diff --git a/tests/unit/test_readme_counts.py b/tests/unit/test_readme_counts.py index 49b4829..e3e28c7 100644 --- a/tests/unit/test_readme_counts.py +++ b/tests/unit/test_readme_counts.py @@ -80,3 +80,32 @@ def test_every_command_named_in_the_table_exists() -> None: named = set(re.findall(r"`apiverity (\w[\w-]*)", README)) unknown = sorted(named - known - {"lab"}) assert not unknown, f"README names commands that do not exist: {unknown}" + + +def test_the_declared_version_is_the_same_everywhere() -> None: + """pyproject, CITATION.cff and the CHANGELOG must agree. + + ToolTrace Bench shipped with these three saying three different things -- + two of them describing a version that had never been tagged. It is a + one-line fix each time and it happens on every release, which is what a + test is for. + """ + import re + import tomllib + + pyproject = tomllib.loads((ROOT / "pyproject.toml").read_text(encoding="utf-8")) + declared = pyproject["project"]["version"] + + citation = (ROOT / "CITATION.cff").read_text(encoding="utf-8") + match = re.search(r"^version:\s*(\S+)$", citation, re.M) + assert match, "CITATION.cff has no version field" + assert match.group(1).strip("'\"") == declared, ( + f"CITATION.cff says {match.group(1)}, pyproject says {declared}" + ) + + changelog = (ROOT / "CHANGELOG.md").read_text(encoding="utf-8") + released = re.findall(r"^## \[(\d+\.\d+\.\d+)\]", changelog, re.M) + assert released, "the CHANGELOG has no released version heading" + assert released[0] == declared, ( + f"the newest CHANGELOG entry is {released[0]}, pyproject says {declared}" + ) From e42a1d0135e707285b1af16814be800e1d73d756 Mon Sep 17 00:00:00 2001 From: Samran Asif Date: Mon, 7 Sep 2026 14:43:09 +0500 Subject: [PATCH 2/2] fix(release): check out in the release job, and guard the PyPI upload Two things that would have made the first tag fail, both found by pushing a tag on a sibling repo rather than by reading the file. `gh release create` shells out to git for the repository context, and the GitHub Release job had no checkout. On local-ai-hardware-bench's first ever tag this produced "fatal: not a git repository" after the build, SBOM and checksums had all succeeded -- no release object, nothing published, the tag simply produced nothing. Same job shape here. ToolTrace Bench's equivalent does check out, which is why it is the only one of these repos that has shipped a release. The PyPI upload was unconditional. It needs a Trusted Publisher registered for this project on pypi.org, which is a form on the account that owns the name and cannot be created from a repository. Without a guard the job fails on every tag even when everything else succeeded, and a red release is indistinguishable from a broken one. Now it skips with a notice naming exactly what to register; set PUBLISH_ENABLED=true once it exists. Co-Authored-By: Claude Opus 5 --- .github/workflows/release.yml | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index a9cd8b3..9319840 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -44,7 +44,25 @@ jobs: with: name: distributions path: dist/ + # Guarded rather than unconditional. Publishing needs a Trusted + # Publisher registered for this project on pypi.org, which cannot be + # created from a repository -- it is a form on the PyPI account that + # owns the name. Without the guard a tagged release fails here even + # though the build, the GitHub Release and the image all succeeded, and + # a red release is indistinguishable from a broken one. + - name: Is Trusted Publishing configured? + id: guard + env: + PUBLISH_ENABLED: ${{ vars.PUBLISH_ENABLED }} + run: | + if [ "${PUBLISH_ENABLED}" = "true" ]; then + echo "enabled=true" >> "$GITHUB_OUTPUT" + else + echo "enabled=false" >> "$GITHUB_OUTPUT" + echo "::notice::Skipping PyPI upload: set the repository variable PUBLISH_ENABLED=true after registering a Trusted Publisher (owner webdevsamran, repo api-verity-lab, workflow release.yml, environment pypi)." + fi - name: Publish + if: steps.guard.outputs.enabled == 'true' uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # v1.14.2 github-release: @@ -54,6 +72,13 @@ jobs: permissions: contents: write steps: + # `gh release create` shells out to git for the repository context. + # Without a checkout it fails with "fatal: not a git repository" -- which + # is exactly what happened on the first tag local-ai-hardware-bench ever + # pushed: build, SBOM and checksums all succeeded and no release was + # created. This job needs no source, only the repo, but it does need + # that. + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 with: name: distributions