Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,25 @@ jobs:
with:
name: distributions
path: dist/
# Guarded rather than unconditional. Publishing needs a Trusted
# Publisher registered for this project on pypi.org, which cannot be
# created from a repository -- it is a form on the PyPI account that
# owns the name. Without the guard a tagged release fails here even
# though the build, the GitHub Release and the image all succeeded, and
# a red release is indistinguishable from a broken one.
- name: Is Trusted Publishing configured?
id: guard
env:
PUBLISH_ENABLED: ${{ vars.PUBLISH_ENABLED }}
run: |
if [ "${PUBLISH_ENABLED}" = "true" ]; then
echo "enabled=true" >> "$GITHUB_OUTPUT"
else
echo "enabled=false" >> "$GITHUB_OUTPUT"
echo "::notice::Skipping PyPI upload: set the repository variable PUBLISH_ENABLED=true after registering a Trusted Publisher (owner webdevsamran, repo api-verity-lab, workflow release.yml, environment pypi)."
fi
- name: Publish
if: steps.guard.outputs.enabled == 'true'
uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # v1.14.2

github-release:
Expand All @@ -54,6 +72,13 @@ jobs:
permissions:
contents: write
steps:
# `gh release create` shells out to git for the repository context.
# Without a checkout it fails with "fatal: not a git repository" -- which
# is exactly what happened on the first tag local-ai-hardware-bench ever
# pushed: build, SBOM and checksums all succeeded and no release was
# created. This job needs no source, only the repo, but it does need
# that.
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: distributions
Expand Down
101 changes: 101 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,107 @@ All notable changes. Format based on Keep a Changelog; versions are semver.

## [Unreleased]

## [0.2.0] - 2026-09-07

Ten open issues closed. The theme running through them is that several
documented capabilities were declared and never wired up, and several rules
reported the opposite of what had happened.

### Added — protocols

- **AsyncAPI 3.x** (#18). The adapter handled 2.x only. Direction is now
normalized to the application's point of view, because the two versions'
words are inverted -- AsyncAPI 2's `publish` describes what the application
*consumes* and `subscribe` what it *produces*, while 3's `send`/`receive`
read from the application's side. Storing the raw word made a 2.x document
and its own 3.x migration compare as two unrelated contracts. The adapter is
also registered under the `apiverity.specs` entry point, which it was not.
- **Compiled gRPC descriptor sets** (#17): `.desc`/`.pb`/`.protoset` load,
decoded from the protobuf wire format with no protobuf runtime dependency.
For a proto with imports this is the only input that can be correct.
- **GraphQL operation testing** (#16): schema-driven query generation,
persisted operation documents via `test --operations`, `{data, errors}`
envelope assertions, and introspection-based drift via `drift --base-url`.
GraphQL cannot go through the HTTP runner -- a server answers a malformed
query with 200 and an `errors` array, so a status-code verdict marks every
failure a pass.

### Added — analysis

- **Corpus drift** (#21): `drift --corpus traffic.har` aggregates findings with
a frequency per operation and separates systematic drift from one-offs. A
thousand entries against a service missing one declared header used to
produce a thousand identical findings.
- **Statistically sound performance gates** (#24): bootstrap confidence
intervals for percentiles and throughput, a Wilson interval for error rate,
`--warmup`, and per-metric `--tolerance`. A change must clear the tolerance
*and* have non-overlapping intervals before it is called a regression; where
they overlap the run is reported inconclusive rather than passed. Measured
against a fixed local target over ten runs, this took asserted regressions
from 22 to 8 -- all of them false by construction, because nothing changed.
- **Pluggable case generators** (#19): the `apiverity.generators` entry point
was declared, documented and never read. Four strategies ship with it --
unicode, nesting, numeric boundaries and defensive header safety.
- **Workflow inference** (#20): `workflow --infer` drafts a manifest from the
`links` a spec declares, and only from those. Every step is emitted
commented out; destructive steps are commented twice, so uncommenting the
file wholesale does not arm them.
- Seven protobuf compatibility rules and the rest of the request/response
requiredness catalog (#15, #17). The catalog is 44 rules, generated.

### Fixed — rules that reported the opposite of what happened

- A field **becoming required** in a request body was reported as
`BRK-PARAM-OPTIONALIZED` at INFO. Requiredness was only compared in one
direction and passed no new value, so a breaking change passed a CI gate as
informational. A response field becoming optional was not detected at all.
- Dropping a required field from an AsyncAPI message the application **sends**
was reported as a request relaxation -- "senders are unaffected" -- when the
application is the sender and the consumers break.
- A **unary gRPC RPC becoming bidirectional** produced zero changes. The
`stream` markers were captured by the parser and discarded.
- A protobuf field **renamed at the same number** is wire-compatible and is now
reported as such at WARN; only a number whose type changed is an error.
Reporting every rename as data corruption trains people to ignore the rule
that catches actual corruption.

### Fixed — checks that were never reached

- `PROTO-FIELD-NUMBER-REUSE` was constructed on every duplicate field number
and then dropped: the function returned only the schema.
- `detect_drift` accepted `forbid_undeclared_fields` and passed a hardcoded
`True`.
- `EXIT_UNREACHABLE` was unreachable. `measure` catches connection errors per
request, so a target with nothing listening produced a clean report and
exit 0.
- `import_har` called `json.loads` on every postData; one HTML error page took
down the import of an entire corpus.

### Fixed — reports

- **SARIF carries locations** (#22). Findings have always had file, line,
column and JSON pointer; the renderer emitted none of it, so GitHub code
scanning annotated the repository rather than the line. Adds rule metadata
and `partialFingerprints` that exclude the message, so an alert is not
retired and re-raised when a number in its text changes.
- **JUnit emits testcases.** It declared `tests="N"` over an empty testsuite,
so every consumer read zero tests.
- **HTML escapes spec content** and is fully self-contained, with filter state
in the URL hash. Messages carry field names straight out of a user's
document and were interpolated raw.
- `apiverity report` no longer carries its own copy of every renderer; the two
had already diverged.

### Changed

- The demo app loads page groups as separate chunks (#23): 224 kB in one file
to a 198 kB entry plus 31 kB across six lazy chunks.
- Docs corrected where they described capabilities that did not exist:
AsyncAPI was "planned" while a 2.x adapter shipped, gRPC descriptor import
was "EXISTING" with nothing in the plugin mentioning descriptors, and
workflow inference was "PARTIAL" with no inference of any kind.
- README count claims are re-derived and tested, not restated.

### Fixed
- **Console script entry point** pointed at a nonexistent symbol
(`apiverity.cli.main:cli`); installing the package produced an `apiverity`
Expand Down
4 changes: 2 additions & 2 deletions CITATION.cff
Original file line number Diff line number Diff line change
Expand Up @@ -21,5 +21,5 @@ keywords:
- breaking-changes
- drift-detection
license: Apache-2.0
version: 0.1.0
date-released: '2026-08-22'
version: 0.2.0
date-released: '2026-09-07'
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "hatchling.build"

[project]
name = "api-verity-lab"
version = "0.1.0"
version = "0.2.0"
description = "Unified API contract governance, breaking-change analysis, schema-driven testing, runtime drift detection, traffic replay and performance regression for OpenAPI, GraphQL and gRPC."
readme = "README.md"
license = { text = "Apache-2.0" }
Expand Down
29 changes: 29 additions & 0 deletions tests/unit/test_readme_counts.py
Original file line number Diff line number Diff line change
Expand Up @@ -80,3 +80,32 @@ def test_every_command_named_in_the_table_exists() -> None:
named = set(re.findall(r"`apiverity (\w[\w-]*)", README))
unknown = sorted(named - known - {"lab"})
assert not unknown, f"README names commands that do not exist: {unknown}"


def test_the_declared_version_is_the_same_everywhere() -> None:
"""pyproject, CITATION.cff and the CHANGELOG must agree.

ToolTrace Bench shipped with these three saying three different things --
two of them describing a version that had never been tagged. It is a
one-line fix each time and it happens on every release, which is what a
test is for.
"""
import re
import tomllib

pyproject = tomllib.loads((ROOT / "pyproject.toml").read_text(encoding="utf-8"))
declared = pyproject["project"]["version"]

citation = (ROOT / "CITATION.cff").read_text(encoding="utf-8")
match = re.search(r"^version:\s*(\S+)$", citation, re.M)
assert match, "CITATION.cff has no version field"
assert match.group(1).strip("'\"") == declared, (
f"CITATION.cff says {match.group(1)}, pyproject says {declared}"
)

changelog = (ROOT / "CHANGELOG.md").read_text(encoding="utf-8")
released = re.findall(r"^## \[(\d+\.\d+\.\d+)\]", changelog, re.M)
assert released, "the CHANGELOG has no released version heading"
assert released[0] == declared, (
f"the newest CHANGELOG entry is {released[0]}, pyproject says {declared}"
)
Loading