Skip to content

[ENHANCEMENT] [MER-5674] Add LotE Plate Tectonics adaptive lesson strict Playwright coverage - #6761

Closed
Francisco-Castro wants to merge 1 commit into
masterfrom
MER-5674-adaptive-lesson-lot-e-plate-tectonics-playwright
Closed

[ENHANCEMENT] [MER-5674] Add LotE Plate Tectonics adaptive lesson strict Playwright coverage#6761
Francisco-Castro wants to merge 1 commit into
masterfrom
MER-5674-adaptive-lesson-lot-e-plate-tectonics-playwright

Conversation

@Francisco-Castro

@Francisco-Castro Francisco-Castro commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

MER-5674

Summary

  • Adds Playwright E2E coverage for the adaptive lesson "Plate Tectonics" (Living on the Edge, Unit 1: Geologic Risk), driving all 22 screens through the happy path
  • Introduces a strict verification path for adaptive lessons: instead of asserting the completion text, the walk verifies per screen that the answer actually registered (readback + deferred-save receipt), the submitted request body carried the intended answer, and the server's own evaluation returned actions.correct: true, with an ordered ledger asserted at the end
  • Why: recon runs reached the lesson end with 4 of 22 screens never answered and the old assertion passed — 3 of 22 screens are authored to navigate away even on a wrong answer, so "reached the end" proves neither answered nor correct
  • The two merged specs (MER-5672/5673) keep their existing compatibility walk unchanged; they benefit only from shared helper fixes (verified fills, no first-option guessing)

What changed

New files:

  • lote-plate-tectonics.spec.ts — the E2E test on the strict entry point (173 lines)
  • AdaptiveStrictContract.ts — manifest validation, receipt/ledger types, payload matchers, assertLedger
  • AdaptiveEvaluationObserver.ts — multi-event evaluation collector; attributes traffic by submitted part-path prefixes (attempt GUIDs rotate legally), separates saves/finalizes/evaluations by shape
  • adaptive-strict-driver.spec.ts, adaptive-strict-walk.spec.ts — 64 stubbed tests (no server): observer negatives, ledger negatives, and a scripted fake deck driving the strict walk through duplicate/foreign/out-of-order traffic

Modified files:

  • AdaptiveHappyPathTask.ts — new completeAdaptiveHappyPathStrict (per-screen manifest, one click per action, explicit transition state machine, navigation licensed via the server-minted attempt chain); the existing completeAdaptiveHappyPath compat walk preserved verbatim
  • AdaptiveDeckPO.ts — screen identity from [model][context] (sequenceId/resourceId/attemptGuid), part inventory, verified widget drivers (fill-in-the-blanks via the widget's own selectmenu — programmatic DOM writes never reach the CAPI model and submit Selected Index: -1; drag-and-drop verified by DOM re-parenting, not geometry)
  • playwright.config.ts — actionTimeout (a flaky iframe once held a getAttribute for 732 s)
  • CHANGELOG.md — skip notice includes MER-5674

Test environment setup

CI does not run these suites yet — MER-5857 tracks provisioning; this is for running locally.

Same env vars as MER-5672/5673 (PLAYWRIGHT_SCENARIO_TOKEN, PLAYWRIGHT_ASSETS_BUCKET, PLAYWRIGHT_AUTOMATION_API_KEY).

Seed the bucket with the MER-5674 private assets (course IP, not in the repo):

<your-bucket>/
└── mer-5674/
    ├── living-on-the-edge-course.zip
    └── answers.json   # strict per-screen manifest (22 screens)

PLAYWRIGHT_BASE_URL must be http://127.0.0.1localhost resolves to ::1 and the archive fetch dies with ECONNREFUSED.

Running locally

cd assets/automation
npx playwright test lote-plate-tectonics      # ~5 min live
npx playwright test adaptive-strict           # 64 stubbed tests, ~17 s, no server

Verification

  • Live strict runs green ×4 (4.5–4.8 min), all 22 screens with verdict=true payloadMatch=true
  • Canary: poisoning one answer fails the run at exactly that screen — the test can fail for the reason it exists
  • Cross-model code review: 9 rounds, final verdict approved; every should-fix applied and re-verified
  • Stubbed suites 64/64; tsc/eslint/prettier clean on touched files (2 pre-existing liveSocket tsc errors excluded, present on clean master)

Known limitations

  • Transition derivation reads results[0]; activities with custom.combineFeedback (4 of these 22 screens) can process several results — runs pass because the derivations agree here; recorded as a limit, addressed by the follow-up framework ticket
  • Teardown leaks the imported project per run — server-side bug, diagnosed in TRIAGE-2419

Follow-up

  • Adaptive Lessons: Strict Verification Framework (ticket TBD, linked to this one) — restructures this machinery into a shared journal/registry/oracle framework and migrates MER-5672/5673 to the strict path; spec already drafted and cross-reviewed (8 design passes)
  • MER-5857 — CI provisioning for archive-backed suites

@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor
Warnings
⚠️ PR is large (3885 LOC changed). Consider splitting.

Risk score: 10 → risk/high

Generated by 🚫 dangerJS against b8042f0

@Francisco-Castro Francisco-Castro self-assigned this Aug 5, 2026
@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

AI Review —

Transition actions from later results are ignored

file: assets/automation/src/systems/torus/tasks/AdaptiveStrictContract.ts
line: 371
Description: deriveTransition reads only actions.results[0]. Multi-part evaluations can place feedback or navigation actions in later results, causing a valid transition to be classified as none.
Suggestion: Flatten the actions from every result before deriving feedback and navigation, e.g. actions.results?.flatMap(result => result.params?.actions ?? []) ?? [].

Unescaped part IDs can invalidate the locator

file: assets/automation/src/systems/torus/pom/delivery/AdaptiveDeckPO.ts
line: 419
Description: Interpolating partId into #${partId} creates an invalid selector when the ID contains CSS-special characters such as :. Because it is part of a selector list, the safe attribute-selector alternative does not prevent the entire selector from failing.
Suggestion: Use only an escaped locator, such as this.page.locator(\#${CSS.escape(partId)}`)`, or construct a safely quoted attribute selector without combining it with an unescaped ID selector.

@Francisco-Castro

Copy link
Copy Markdown
Contributor Author

Superseded by #6788, which carries this branch commit for commit plus the shared strict verification framework (MER-5865).

Why close instead of merge: every commit of this PR's branch is an ancestor of #6788's HEAD (git merge-base --is-ancestor confirms it), so nothing here is lost. The LotE spec has since moved from the self-graded strict walker to an audited run boundary whose pass condition is an independent oracle over wire evidence, so merging this first would land a spec that #6788 immediately rewrites — and pay its master conflicts twice.

MER-5674 is delivered by #6788, whose title carries both tickets.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant