diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index a9cd8b3..9319840 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -44,7 +44,25 @@ jobs: with: name: distributions path: dist/ + # Guarded rather than unconditional. Publishing needs a Trusted + # Publisher registered for this project on pypi.org, which cannot be + # created from a repository -- it is a form on the PyPI account that + # owns the name. Without the guard a tagged release fails here even + # though the build, the GitHub Release and the image all succeeded, and + # a red release is indistinguishable from a broken one. + - name: Is Trusted Publishing configured? + id: guard + env: + PUBLISH_ENABLED: ${{ vars.PUBLISH_ENABLED }} + run: | + if [ "${PUBLISH_ENABLED}" = "true" ]; then + echo "enabled=true" >> "$GITHUB_OUTPUT" + else + echo "enabled=false" >> "$GITHUB_OUTPUT" + echo "::notice::Skipping PyPI upload: set the repository variable PUBLISH_ENABLED=true after registering a Trusted Publisher (owner webdevsamran, repo api-verity-lab, workflow release.yml, environment pypi)." + fi - name: Publish + if: steps.guard.outputs.enabled == 'true' uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # v1.14.2 github-release: @@ -54,6 +72,13 @@ jobs: permissions: contents: write steps: + # `gh release create` shells out to git for the repository context. + # Without a checkout it fails with "fatal: not a git repository" -- which + # is exactly what happened on the first tag local-ai-hardware-bench ever + # pushed: build, SBOM and checksums all succeeded and no release was + # created. This job needs no source, only the repo, but it does need + # that. + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 with: name: distributions diff --git a/CHANGELOG.md b/CHANGELOG.md index 16d0949..8ce8e43 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,107 @@ All notable changes. Format based on Keep a Changelog; versions are semver. ## [Unreleased] +## [0.2.0] - 2026-09-07 + +Ten open issues closed. The theme running through them is that several +documented capabilities were declared and never wired up, and several rules +reported the opposite of what had happened. + +### Added — protocols + +- **AsyncAPI 3.x** (#18). The adapter handled 2.x only. Direction is now + normalized to the application's point of view, because the two versions' + words are inverted -- AsyncAPI 2's `publish` describes what the application + *consumes* and `subscribe` what it *produces*, while 3's `send`/`receive` + read from the application's side. Storing the raw word made a 2.x document + and its own 3.x migration compare as two unrelated contracts. The adapter is + also registered under the `apiverity.specs` entry point, which it was not. +- **Compiled gRPC descriptor sets** (#17): `.desc`/`.pb`/`.protoset` load, + decoded from the protobuf wire format with no protobuf runtime dependency. + For a proto with imports this is the only input that can be correct. +- **GraphQL operation testing** (#16): schema-driven query generation, + persisted operation documents via `test --operations`, `{data, errors}` + envelope assertions, and introspection-based drift via `drift --base-url`. + GraphQL cannot go through the HTTP runner -- a server answers a malformed + query with 200 and an `errors` array, so a status-code verdict marks every + failure a pass. + +### Added — analysis + +- **Corpus drift** (#21): `drift --corpus traffic.har` aggregates findings with + a frequency per operation and separates systematic drift from one-offs. A + thousand entries against a service missing one declared header used to + produce a thousand identical findings. +- **Statistically sound performance gates** (#24): bootstrap confidence + intervals for percentiles and throughput, a Wilson interval for error rate, + `--warmup`, and per-metric `--tolerance`. A change must clear the tolerance + *and* have non-overlapping intervals before it is called a regression; where + they overlap the run is reported inconclusive rather than passed. Measured + against a fixed local target over ten runs, this took asserted regressions + from 22 to 8 -- all of them false by construction, because nothing changed. +- **Pluggable case generators** (#19): the `apiverity.generators` entry point + was declared, documented and never read. Four strategies ship with it -- + unicode, nesting, numeric boundaries and defensive header safety. +- **Workflow inference** (#20): `workflow --infer` drafts a manifest from the + `links` a spec declares, and only from those. Every step is emitted + commented out; destructive steps are commented twice, so uncommenting the + file wholesale does not arm them. +- Seven protobuf compatibility rules and the rest of the request/response + requiredness catalog (#15, #17). The catalog is 44 rules, generated. + +### Fixed — rules that reported the opposite of what happened + +- A field **becoming required** in a request body was reported as + `BRK-PARAM-OPTIONALIZED` at INFO. Requiredness was only compared in one + direction and passed no new value, so a breaking change passed a CI gate as + informational. A response field becoming optional was not detected at all. +- Dropping a required field from an AsyncAPI message the application **sends** + was reported as a request relaxation -- "senders are unaffected" -- when the + application is the sender and the consumers break. +- A **unary gRPC RPC becoming bidirectional** produced zero changes. The + `stream` markers were captured by the parser and discarded. +- A protobuf field **renamed at the same number** is wire-compatible and is now + reported as such at WARN; only a number whose type changed is an error. + Reporting every rename as data corruption trains people to ignore the rule + that catches actual corruption. + +### Fixed — checks that were never reached + +- `PROTO-FIELD-NUMBER-REUSE` was constructed on every duplicate field number + and then dropped: the function returned only the schema. +- `detect_drift` accepted `forbid_undeclared_fields` and passed a hardcoded + `True`. +- `EXIT_UNREACHABLE` was unreachable. `measure` catches connection errors per + request, so a target with nothing listening produced a clean report and + exit 0. +- `import_har` called `json.loads` on every postData; one HTML error page took + down the import of an entire corpus. + +### Fixed — reports + +- **SARIF carries locations** (#22). Findings have always had file, line, + column and JSON pointer; the renderer emitted none of it, so GitHub code + scanning annotated the repository rather than the line. Adds rule metadata + and `partialFingerprints` that exclude the message, so an alert is not + retired and re-raised when a number in its text changes. +- **JUnit emits testcases.** It declared `tests="N"` over an empty testsuite, + so every consumer read zero tests. +- **HTML escapes spec content** and is fully self-contained, with filter state + in the URL hash. Messages carry field names straight out of a user's + document and were interpolated raw. +- `apiverity report` no longer carries its own copy of every renderer; the two + had already diverged. + +### Changed + +- The demo app loads page groups as separate chunks (#23): 224 kB in one file + to a 198 kB entry plus 31 kB across six lazy chunks. +- Docs corrected where they described capabilities that did not exist: + AsyncAPI was "planned" while a 2.x adapter shipped, gRPC descriptor import + was "EXISTING" with nothing in the plugin mentioning descriptors, and + workflow inference was "PARTIAL" with no inference of any kind. +- README count claims are re-derived and tested, not restated. + ### Fixed - **Console script entry point** pointed at a nonexistent symbol (`apiverity.cli.main:cli`); installing the package produced an `apiverity` diff --git a/CITATION.cff b/CITATION.cff index 62664ec..19cd28a 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -21,5 +21,5 @@ keywords: - breaking-changes - drift-detection license: Apache-2.0 -version: 0.1.0 -date-released: '2026-08-22' \ No newline at end of file +version: 0.2.0 +date-released: '2026-09-07' \ No newline at end of file diff --git a/pyproject.toml b/pyproject.toml index d0101ea..c20784a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "api-verity-lab" -version = "0.1.0" +version = "0.2.0" description = "Unified API contract governance, breaking-change analysis, schema-driven testing, runtime drift detection, traffic replay and performance regression for OpenAPI, GraphQL and gRPC." readme = "README.md" license = { text = "Apache-2.0" } diff --git a/tests/unit/test_readme_counts.py b/tests/unit/test_readme_counts.py index 49b4829..e3e28c7 100644 --- a/tests/unit/test_readme_counts.py +++ b/tests/unit/test_readme_counts.py @@ -80,3 +80,32 @@ def test_every_command_named_in_the_table_exists() -> None: named = set(re.findall(r"`apiverity (\w[\w-]*)", README)) unknown = sorted(named - known - {"lab"}) assert not unknown, f"README names commands that do not exist: {unknown}" + + +def test_the_declared_version_is_the_same_everywhere() -> None: + """pyproject, CITATION.cff and the CHANGELOG must agree. + + ToolTrace Bench shipped with these three saying three different things -- + two of them describing a version that had never been tagged. It is a + one-line fix each time and it happens on every release, which is what a + test is for. + """ + import re + import tomllib + + pyproject = tomllib.loads((ROOT / "pyproject.toml").read_text(encoding="utf-8")) + declared = pyproject["project"]["version"] + + citation = (ROOT / "CITATION.cff").read_text(encoding="utf-8") + match = re.search(r"^version:\s*(\S+)$", citation, re.M) + assert match, "CITATION.cff has no version field" + assert match.group(1).strip("'\"") == declared, ( + f"CITATION.cff says {match.group(1)}, pyproject says {declared}" + ) + + changelog = (ROOT / "CHANGELOG.md").read_text(encoding="utf-8") + released = re.findall(r"^## \[(\d+\.\d+\.\d+)\]", changelog, re.M) + assert released, "the CHANGELOG has no released version heading" + assert released[0] == declared, ( + f"the newest CHANGELOG entry is {released[0]}, pyproject says {declared}" + )