chore(e2e): resolve expected tags/digests from the registries at runtime; migrate the harness to TypeScript - #1121
Open
avargaskun wants to merge 2 commits into
Open
chore(e2e): resolve expected tags/digests from the registries at runtime; migrate the harness to TypeScript#1121avargaskun wants to merge 2 commits into
avargaskun wants to merge 2 commits into
Conversation
Contributor
Author
|
@fmartinou The motivation for this change: every time an image in the e2e fixtures published a new tag, the hardcoded expectations went stale and the suite went red — I see the same churn on your side too (the All 42 scenarios pass unchanged — only the expectations became dynamic, no assertions were removed or weakened. |
…e for dynamic version assertions Replace hardcoded expected tags/digests in the e2e feature tables with a registry oracle that resolves the latest matching tag or digest live from each registry (hub, ghcr, gitlab, lscr, quay), ending the tag-refresh churn every new upstream image release causes. Digests are resolved for the host architecture so the suite passes on arm64 and amd64 hosts alike. Also run the UI unit tests serially (--runInBand) to avoid jest worker crashes on memory-constrained runners.
avargaskun
force-pushed
the
feat/e2e-ts-oracle
branch
from
July 27, 2026 02:54
a357d59 to
d0d0810
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The e2e feature tables currently hardcode the tag and digest that WUD is expected to
report as the latest available version for each watched image. Those expectations go
stale every time one of the watched upstream images publishes a release, so the suite
has to be repaired on a recurring basis — 4e1e3b0 and 7647bcb ("Fix E2E tests",
2026-07-18) are the two most recent instances, and the pattern goes back further.
This PR removes the cause rather than refreshing the values again. The expected tag or
digest is resolved at test time from the same registry WUD itself queries, using a
per-row
strategyandpattern. The assertions themselves are unchanged: the suitestill asserts that
result.tag/result.digest/updateAvailablehave specificvalues — the values are now derived instead of transcribed.
What the oracle does
e2e/features/support/registry_oracle.tsexposes two functions:getLatestVersion(registry, image, pattern)— lists tags from the registry(Docker Hub v2 API, GHCR/LSCR token +
/v2/.../tags/listwithLink-headerpagination, GitLab JWT + registry.gitlab.com, Quay
/api/v1/repository/.../tag/),filters them by
pattern, sorts withsemver(falling back tosemver.coerceand then lexical comparison for non-semver tags), and returns the highest.
getLatestDigest(registry, image, tag)— fetches the manifest, and for a manifestlist / OCI index picks the entry matching the relevant architecture, falling back
to a
HEADrequest for thedocker-content-digestheader on single manifests.Each row in
api-container.featureandprometheus.featuregained two columns:dynamicpatternstaticresultTagcolumnstaticis used where the expectation is genuinely fixed (latestrows, and thecommented-out ECR row), so no row loses its assertion.
patternencodes the updatesemantics the row is actually testing, e.g.:
^\d+\.\d+-alpine$— the nginx alpine minor-update row^\d+\.\d+\.\d+\.\d+-ls\d+$— the linuxserver/radarr complex-semver rows^v16\.[01]\.0$— the GitLab row, bounded so it keeps testing a minor updaterather than tracking gitlab-runner's tip forever
^v\d+\.\d+\.\d+$— the Quay prometheus major-update rowThe steps that resolve and compare are in
e2e/features/step_definitions/custom_steps.ts(six generic steps plus abacktick-variable substitution helper for the Prometheus text-body assertions).
Host-architecture digest resolution
Digest assertions previously only matched on whichever architecture the manifest
list happened to yield, so the digest scenarios failed on arm64 machines. The oracle
now inspects the locally pulled image (
docker image inspect --format '{{.Architecture}}') and selects the manifest for that architecture, falling back tothe host architecture derived from
os.arch(). The suite passes on both arm64 andamd64 hosts.
TypeScript migration
The harness (not the application) moves to TypeScript so the oracle and the new step
definitions are type-checked:
e2e/config/index.js→index.ts,e2e/features/support/init.js→init.ts,e2e/features/step_definitions/api.js→api.tse2e/tsconfig.jsonadded (strict, commonjs,types: ["node", "semver"])ts-node,typescript,@types/node,@types/semveradded as devDependencies;semveras a dependencycucumberscript runscucumber-js --require-module ts-node/register --require 'features/**/*.ts'@cucumber/cucumbermoved from11.1.1to^11.3.0(needed for the typings usedby the step definitions)
now involves registry round-trips
npx tsc --noEmitis clean andcucumber-js --dry-runbinds all 397 steps across all42 scenarios.
Serial jest for the UI unit tests
ui/package.jsongains a singletestscript that runs jest with--runInBandandwithout coverage, to avoid jest worker crashes on memory-constrained runners. This is
a pure addition:
test:unitandtest:unit:watchare untouched, so the existing.travis.ymlstep (cd ui && npm run test:unit) keeps its current coverage andparallel semantics.
e2e lint script
The
lintscript becomeseslint '**/*.ts'. The'**/*.js'glob is dropped becausethe migration leaves no
.jsfiles undere2e/, and ESLint 8 exits non-zero on aglob that matches nothing.
e2e/.eslintrcgains the TypeScript parser, plugin andimport resolver needed to lint the migrated sources under the existing
airbnb-baseconfig; see the limitations section for details.
npm run lintexits 0.npm test aliases
The first commit adds plain
testaliases sonpm testworks frome2e/andui/the same way it already does from
app/.Validation
set with no scenarios added, removed, or skipped.
npx tsc --noEmit: 0 errors.npx cucumber-js --dry-run: 42 scenarios / 397 steps, no undefined or ambiguous steps.ui:npm run test:unitandnpm testboth green (26 suites, 157 tests).Note: the lint cleanup described below reformatted
registry_oracle.tsafter the 42/42run (style-only — a callback parameter rename and a block-bodied arrow, no logic change).
tscand the cucumber dry-run were re-run clean afterwards, but the full suite should bere-run before merge to confirm.
Limitations / things to review
Network dependency: the oracle performs registry API calls during the test run.
The suite already required registry access (WUD itself queries the same registries
for these scenarios), so this adds no new class of dependency, but it does add
per-scenario HTTP calls and therefore some latency and a new failure mode if a
registry API is rate-limiting or down. The step timeout was raised to 60s partly for
this reason.
Anonymous rate limits: Docker Hub and GHCR token endpoints are called
unauthenticated. On a busy shared CI IP this could hit anonymous rate limits.
Pattern maintenance: the patterns are now the thing that encodes intent. A row
whose pattern is too loose would silently start tracking a different update than it
was written to test (this is why the GitLab row is bounded to
^v16\.[01]\.0$rather than open-ended).
ECR row: still commented out upstream; its
staticstrategy preserves theoriginal literal so nothing changes if it is re-enabled.
@cucumber/cucumberpin loosened from the exact11.1.1to^11.3.0. If theproject prefers exact pins in
e2e/, say so and it can be pinned to a specific11.3.x.
apickliis unchanged at3.0.3.e2e/package-lock.jsonis fully regenerated, so the diff is large and includesunrelated transitive churn. Happy to redo it however the project prefers.
TypeScript linting is newly enabled, which is why
e2e/.eslintrcand the lintdevDependencies grew. Previously
eslint '**/*.js'failed on an unmatched glob beforeit ever parsed anything, so the TS sources were effectively unlinted. Enabling it
required
@typescript-eslint/parser(parse TS at all),@typescript-eslint/eslint-pluginwith the standardno-unused-varsswap (the baserule flags TypeScript
thisparameters as unused), andeslint-import-resolver-typescriptplusimport/extensionsset toneverfor.ts(so
eslint-plugin-importcan resolve extensionless TS imports). All the style issuesthis surfaced are fixed in the source rather than silenced. The only suppressions are
three documented inline
eslint-disable-next-linecomments on the registry oracle'stwo pagination loops (
no-await-in-loop×2,no-loop-func): paging is necessarilysequential, since each request's URL or continuation flag is only known once the
previous response has been read.
npm run lintexits 0; tenfunc-names/no-consolewarnings remain, which do not affect exit status.(For reference, the
cucumberscript's newdotenvx run -f ../.envprefix is safe onCI: dotenvx warns
MISSING_ENV_FILEbut exits 0 and passes the ambient environmentthrough, which is how Travis already supplies the registry credentials.)