Skip to content

chore(e2e): resolve expected tags/digests from the registries at runtime; migrate the harness to TypeScript - #1121

Open
avargaskun wants to merge 2 commits into
getwud:mainfrom
avargaskun:feat/e2e-ts-oracle
Open

chore(e2e): resolve expected tags/digests from the registries at runtime; migrate the harness to TypeScript#1121
avargaskun wants to merge 2 commits into
getwud:mainfrom
avargaskun:feat/e2e-ts-oracle

Conversation

@avargaskun

Copy link
Copy Markdown
Contributor

Summary

The e2e feature tables currently hardcode the tag and digest that WUD is expected to
report as the latest available version for each watched image. Those expectations go
stale every time one of the watched upstream images publishes a release, so the suite
has to be repaired on a recurring basis — 4e1e3b0 and 7647bcb ("Fix E2E tests",
2026-07-18) are the two most recent instances, and the pattern goes back further.

This PR removes the cause rather than refreshing the values again. The expected tag or
digest is resolved at test time from the same registry WUD itself queries, using a
per-row strategy and pattern. The assertions themselves are unchanged: the suite
still asserts that result.tag / result.digest / updateAvailable have specific
values — the values are now derived instead of transcribed.

What the oracle does

e2e/features/support/registry_oracle.ts exposes two functions:

  • getLatestVersion(registry, image, pattern) — lists tags from the registry
    (Docker Hub v2 API, GHCR/LSCR token + /v2/.../tags/list with Link-header
    pagination, GitLab JWT + registry.gitlab.com, Quay /api/v1/repository/.../tag/),
    filters them by pattern, sorts with semver (falling back to semver.coerce
    and then lexical comparison for non-semver tags), and returns the highest.
  • getLatestDigest(registry, image, tag) — fetches the manifest, and for a manifest
    list / OCI index picks the entry matching the relevant architecture, falling back
    to a HEAD request for the docker-content-digest header on single manifests.

Each row in api-container.feature and prometheus.feature gained two columns:

strategy meaning
dynamic resolve the expectation from the registry using pattern
static keep the literal value in the resultTag column

static is used where the expectation is genuinely fixed (latest rows, and the
commented-out ECR row), so no row loses its assertion. pattern encodes the update
semantics the row is actually testing, e.g.:

  • ^\d+\.\d+-alpine$ — the nginx alpine minor-update row
  • ^\d+\.\d+\.\d+\.\d+-ls\d+$ — the linuxserver/radarr complex-semver rows
  • ^v16\.[01]\.0$ — the GitLab row, bounded so it keeps testing a minor update
    rather than tracking gitlab-runner's tip forever
  • ^v\d+\.\d+\.\d+$ — the Quay prometheus major-update row

The steps that resolve and compare are in
e2e/features/step_definitions/custom_steps.ts (six generic steps plus a
backtick-variable substitution helper for the Prometheus text-body assertions).

Host-architecture digest resolution

Digest assertions previously only matched on whichever architecture the manifest
list happened to yield, so the digest scenarios failed on arm64 machines. The oracle
now inspects the locally pulled image (docker image inspect --format '{{.Architecture}}') and selects the manifest for that architecture, falling back to
the host architecture derived from os.arch(). The suite passes on both arm64 and
amd64 hosts.

TypeScript migration

The harness (not the application) moves to TypeScript so the oracle and the new step
definitions are type-checked:

  • e2e/config/index.jsindex.ts, e2e/features/support/init.jsinit.ts,
    e2e/features/step_definitions/api.jsapi.ts
  • e2e/tsconfig.json added (strict, commonjs, types: ["node", "semver"])
  • ts-node, typescript, @types/node, @types/semver added as devDependencies;
    semver as a dependency
  • the cucumber script runs cucumber-js --require-module ts-node/register --require 'features/**/*.ts'
  • @cucumber/cucumber moved from 11.1.1 to ^11.3.0 (needed for the typings used
    by the step definitions)
  • the default step timeout was raised from 20s to 60s, since resolving expectations
    now involves registry round-trips

npx tsc --noEmit is clean and cucumber-js --dry-run binds all 397 steps across all
42 scenarios.

Serial jest for the UI unit tests

ui/package.json gains a single test script that runs jest with --runInBand and
without coverage, to avoid jest worker crashes on memory-constrained runners. This is
a pure addition: test:unit and test:unit:watch are untouched, so the existing
.travis.yml step (cd ui && npm run test:unit) keeps its current coverage and
parallel semantics.

e2e lint script

The lint script becomes eslint '**/*.ts'. The '**/*.js' glob is dropped because
the migration leaves no .js files under e2e/, and ESLint 8 exits non-zero on a
glob that matches nothing. e2e/.eslintrc gains the TypeScript parser, plugin and
import resolver needed to lint the migrated sources under the existing airbnb-base
config; see the limitations section for details. npm run lint exits 0.

npm test aliases

The first commit adds plain test aliases so npm test works from e2e/ and ui/
the same way it already does from app/.


Validation

  • Full backend e2e suite: 42/42 scenarios green, run against upstream's own scenario
    set with no scenarios added, removed, or skipped.
  • npx tsc --noEmit: 0 errors.
  • npx cucumber-js --dry-run: 42 scenarios / 397 steps, no undefined or ambiguous steps.
  • ui: npm run test:unit and npm test both green (26 suites, 157 tests).
  • Assertions are preserved one-for-one; only the source of the expected value changed.

Note: the lint cleanup described below reformatted registry_oracle.ts after the 42/42
run (style-only — a callback parameter rename and a block-bodied arrow, no logic change).
tsc and the cucumber dry-run were re-run clean afterwards, but the full suite should be
re-run before merge to confirm.


Limitations / things to review

  • Network dependency: the oracle performs registry API calls during the test run.
    The suite already required registry access (WUD itself queries the same registries
    for these scenarios), so this adds no new class of dependency, but it does add
    per-scenario HTTP calls and therefore some latency and a new failure mode if a
    registry API is rate-limiting or down. The step timeout was raised to 60s partly for
    this reason.

  • Anonymous rate limits: Docker Hub and GHCR token endpoints are called
    unauthenticated. On a busy shared CI IP this could hit anonymous rate limits.

  • Pattern maintenance: the patterns are now the thing that encodes intent. A row
    whose pattern is too loose would silently start tracking a different update than it
    was written to test (this is why the GitLab row is bounded to ^v16\.[01]\.0$
    rather than open-ended).

  • ECR row: still commented out upstream; its static strategy preserves the
    original literal so nothing changes if it is re-enabled.

  • @cucumber/cucumber pin loosened from the exact 11.1.1 to ^11.3.0. If the
    project prefers exact pins in e2e/, say so and it can be pinned to a specific
    11.3.x. apickli is unchanged at 3.0.3.

  • e2e/package-lock.json is fully regenerated, so the diff is large and includes
    unrelated transitive churn. Happy to redo it however the project prefers.

  • TypeScript linting is newly enabled, which is why e2e/.eslintrc and the lint
    devDependencies grew. Previously eslint '**/*.js' failed on an unmatched glob before
    it ever parsed anything, so the TS sources were effectively unlinted. Enabling it
    required @typescript-eslint/parser (parse TS at all),
    @typescript-eslint/eslint-plugin with the standard no-unused-vars swap (the base
    rule flags TypeScript this parameters as unused), and
    eslint-import-resolver-typescript plus import/extensions set to never for .ts
    (so eslint-plugin-import can resolve extensionless TS imports). All the style issues
    this surfaced are fixed in the source rather than silenced. The only suppressions are
    three documented inline eslint-disable-next-line comments on the registry oracle's
    two pagination loops (no-await-in-loop ×2, no-loop-func): paging is necessarily
    sequential, since each request's URL or continuation flag is only known once the
    previous response has been read. npm run lint exits 0; ten func-names /
    no-console warnings remain, which do not affect exit status.

(For reference, the cucumber script's new dotenvx run -f ../.env prefix is safe on
CI: dotenvx warns MISSING_ENV_FILE but exits 0 and passes the ambient environment
through, which is how Travis already supplies the registry credentials.)

@avargaskun

Copy link
Copy Markdown
Contributor Author

@fmartinou The motivation for this change: every time an image in the e2e fixtures published a new tag, the hardcoded expectations went stale and the suite went red — I see the same churn on your side too (the
recurring "Fix E2E tests" commits). With the oracle resolving expected tags/digests from the registries at runtime, the suite stays green without those manual refreshes. Since making this change I've had automated PR
validation running on my fork's dev branch with no test maintenance needed.

All 42 scenarios pass unchanged — only the expectations became dynamic, no assertions were removed or weakened.

…e for dynamic version assertions

Replace hardcoded expected tags/digests in the e2e feature tables with a
registry oracle that resolves the latest matching tag or digest live from
each registry (hub, ghcr, gitlab, lscr, quay), ending the tag-refresh
churn every new upstream image release causes. Digests are resolved for
the host architecture so the suite passes on arm64 and amd64 hosts alike.
Also run the UI unit tests serially (--runInBand) to avoid jest worker
crashes on memory-constrained runners.
@avargaskun
avargaskun force-pushed the feat/e2e-ts-oracle branch from a357d59 to d0d0810 Compare July 27, 2026 02:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant