Skip to content

test: Playwright browser E2E suite on a fully sandboxed stack - #254

Open
kovtcharov wants to merge 7 commits into
mainfrom
test/e2e-playwright
Open

kovtcharov wants to merge 7 commits into
mainfrom
test/e2e-playwright

Conversation

@kovtcharov

@kovtcharov kovtcharov commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

What

A browser-level end-to-end suite: real built backend + real built frontend bundle + real Chromium, driven by Playwright, against a fully sandboxed Claudia stack with a fake claude CLI on PATH. 29 specs, zero retries, one worker.

Re-verified under always-on auth (rebased onto 8ce9aa9)

Since this was last green, main landed #261 (a token on every /api route and every WebSocket upgrade), plus #259, #262, #258, #249, #255, #272, #252 and #251. This branch is rebased onto that and re-verified:

  • Conflict 1: frontend/src/config/api-config.ts was rewritten by feat(auth): require a token on every API route and WebSocket upgrade #261 for auth; this PR edits it for VITE_CLAUDIA_BACKEND_PORT. The resolution keeps both: getBackendPort() drives the web HTTP and WS URLs, and the WS URL still goes through feat(auth): require a token on every API route and WebSocket upgrade #261's withWsToken().
  • Conflict 2: fix(data-dir): honor CLAUDIA_DATA_DIR for todos, checkpoints, and shared-MCP pid files #251 landed the same TodoStore(dataDir) / CheckpointStore(dataDir) fix as this PR's fix(server) commit. server.ts keeps main's (identical) code plus this PR's explanatory comment. data-dir-wiring.test.ts keeps both test sets: fix(data-dir): honor CLAUDIA_DATA_DIR for todos, checkpoints, and shared-MCP pid files #251's behavioural tests (boot createApp, prove the files land in the data dir) and this PR's structural guards (no basePath read after ensureDataDir, no argument-less new XStore()), which catch the next store wired up wrong. 10/10 pass.
  • Direct API calls authenticate the way the backend does: authHeaders() in e2e/harness/env.ts reads the token with the backend's own getAuthToken(STATE_DIR) (backend/src/auth-token.ts) against the sandbox's CLAUDIA_DATA_DIR. The harness mints it before the backend boots, so the two processes can never race to mint different tokens.
  • The browser is not given a token. It authenticates through the app's real loopback bootstrap (/api/auth/local), exactly as a user who opens a browser on the same machine would.
  • New assertions:
    • 00-isolation: an unauthenticated request, and one with a well-formed but wrong token, get 401; the sandbox token gets 200, and auth-token lives in the sandbox state dir.
    • 01-app-loads: /api/auth/local answered 200 from the sandboxed backend, the token gate never appears, and every WebSocket URL carries ?token=.
  • Readiness probe: webServer now waits on /api/health (unauthenticated). /api/tasks returns 401 now; Playwright counts a 401 as "up", so the old probe was passing by accident of that rule.
  • Isolation list: auth-token, mcp-token, instance.json and task-histories added to the legacy backend/ locations the sandbox must not write.
  • Run-scoped legacy check: the spec used to assert those locations were absent, which fails on any checkout where other tooling left a file there — the backend unit suite does not set CLAUDIA_DATA_DIR, so a normal npm test -w backend mints backend/mcp-token. prepareHarness() now snapshots each location (mtime, or null) before the backend boots, and the spec asserts the run neither created nor modified any of them. On a fresh checkout (CI) the baseline is all-null, so this is exactly the old "never created" check.
  • Windows safety: on Windows, HOME does not relocate os.homedir() and resolveClaudeSpawn() finds the CLI through APPDATA/USERPROFILE, not PATH. The harness now points both into the sandbox and puts a refusing claude.cmd first on PATH, and uses the platform PATH delimiter. A Windows run can therefore neither write into the real profile nor reach the real Claude CLI. The fake CLI itself is a bash script, so the task specs still need a POSIX host.

Evidence

Local (Windows 11, the specs that do not spawn the fake CLI), zero retries:

Running 19 tests using 1 worker
  ✓ 00-isolation ×4   ✓ 01-app-loads ×4   ✓ 02-workspace-lifecycle ×4
  ✓ 06-settings ×3    ✓ 07-file-explorer ×4
  19 passed (17.6s)   — with a unit-suite-leaked backend/mcp-token present, which the baseline check correctly ignores
after: 4801/5801 free; ~/.claudia-e2e-4801 removed by teardown; no stray backend/preview processes

03-task-lifecycle, 04-reload-reconnect and 05-terminal-resize (10 specs) drive the bash fake CLI and were not run locally: no POSIX host was available on the verifying machine. e2e.yml runs on workflow_dispatch + nightly only, and both require the workflow to be on main, so those 10 specs are first exercised in CI after merge unless a pull_request trigger is added.

Gates on the same tree:

npm test -w backend    2349 passed, 277 skipped (Windows-skipped fake-CLI integration tests)
npm test -w frontend   1153 passed
tsc --noEmit     backend OK · frontend OK · tsconfig.e2e.json OK

Coverage (8 spec areas)

# Spec What it proves
00 isolation Sandboxed ports ≠ 4001/5173; backend reads/writes only the sandbox state dir; legacy backend/* state, token and lock files and the shared-MCP pid file were never created; backend answers on the sandboxed port; /api refuses missing/wrong tokens with 401
01 app-loads Shell renders, WebSocket reaches connected, zero console/page errors on boot, browser authenticates via the loopback bootstrap with a token-bearing WS, browser never contacts 4001/5173
02 workspace-lifecycle Add by path → inline rename → survives reload → remove (via window.confirm) → stays gone after reload
03 task-lifecycle Create task → prompt reaches the fake CLI's stdin at the real process boundary → output painted into xterm → follow-up input delivered → archive persists across reload
04 reload-reconnect Running task survives a full page reload with history replayed; reattached terminal still writes to the live PTY; repeated reloads don't duplicate history
05 terminal-resize ≤2-col wobble sends no task:resize; a large resize sends exactly one; content survives resize; output during the 250ms post-resize buffer still lands
06 settings Skip-permissions toggle round-trips through PUT /api/config and survives reload (both directions); theme persists via localStorage bootstrap
07 file-explorer Tree lists fixture files; lazy directory expand; double-click opens README (markdown) and a nested text file with the fixture's marker

Isolation guarantees — and how each is self-asserted

The suite once contaminated the developer's live instance on 4001. Every guard below is checked at runtime, not assumed:

  • Ports: assertSafePort() hard-fails on 4001/5173; assertPortFree() refuses to attach to an existing listener on 4801/5801.
  • Bundle: the built frontend is scanned for the sandboxed port before any test runs (assertBundleTargetsSandbox) — a bundle built without VITE_CLAUDIA_BACKEND_PORT silently falls back to 4001.
  • In-page guard: fetch, XMLHttpRequest.open and WebSocket throw on :4001/:5173; a per-test fixture also asserts every WebSocket targeted the sandboxed backend.
  • State: CLAUDIA_DATA_DIR + fake HOME (and, on Windows, USERPROFILE/APPDATA) under ~/.claudia-e2e-<port> (under homedir(), not os.tmpdir() — on macOS /tmp resolves under /var, which validateWorkspacePath blocklists).
  • Credentials: the sandbox token lives in the sandbox state dir; the browser obtains it only through the app's real bootstrap.
  • Processes: the only claude reachable is the fake fixture; teardown pkills only processes whose cmdline contains the sandbox's own fake-CLI path. CLAUDIA_SHARED_MCP=0 so the sandboxed backend never probes-and-adopts the developer's shared Playwright MCP server on 4022.

Fixed-port caveat

Ports are fixed at 4801/5801, overridable via CLAUDIA_E2E_BACKEND_PORT / CLAUDIA_E2E_FRONTEND_PORT, and guarded three ways (forbidden-list, free-check, in-page guard). The state root is keyed by the backend port, so two suites on different ports do not share state.

Deliberate bounded wait

There is exactly one waitForTimeout in the suite (05-terminal-resize): the property under test is the absence of a task:resize frame after a 10px wobble, and absence can only be observed by outlasting the 150ms debounce window. Every other wait is a web-first locator/expect.poll.

Product-code changes

Purely additive: data-testid / data-* attributes across components, an optional panelId prop on CollapsiblePanel, data-ws-connected on the app root, a build-time VITE_CLAUDIA_BACKEND_PORT override in api-config.ts (no runtime/query-param seam), and checkpoints/todos kept inside CLAUDIA_DATA_DIR. .gitignore: node_modules/node_modules (in a worktree it is a symlink, which the trailing-slash form did not match).

CI

.github/workflows/e2e.yml: workflow_dispatch + nightly schedule only, npm ci, npx playwright install --with-deps chromium, artifacts on failure. Not a required check and does not run on pull requests; promote to pull_request: once nightly history is green.

Run locally

npx playwright install chromium
npm run test:e2e            # builds shared/backend/frontend, boots the sandbox, runs
npm run test:e2e:headed     # watch it
CLAUDIA_E2E_SKIP_BUILD=1 npx playwright test e2e/tests/05-terminal-resize.spec.ts   # iterate on one spec
CLAUDIA_E2E_KEEP_STATE=1 …  # keep ~/.claudia-e2e-<port> for post-mortem

@kovtcharov kovtcharov left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FIXED — two real sandbox escapes found and fixed on the branch; everything else holds up

Independent adversarial review, no prior context. Built and ran the suite against a real backend + real Chromium, with origin/fix/shared-esm-js-extension (#248) merged locally as the PR instructs. Three commits pushed to this branch; the #248 merge is deliberately not included.

The design is genuinely good — assertSafePort, assertPortFree, the in-page fetch/XHR/WebSocket guard, the bundle-port precondition, CLAUDIA_SHARED_MCP=0, and retries: 0 are all the right calls, and I verified each of them actually fires. But the suite was contaminating the checkout it ran in, and its own isolation spec was structurally incapable of noticing.


🔴 1. The sandboxed backend wrote into the developer's checkout on every run

~/.claudia-e2e/... is honoured for most stores — but not all. After every green run:

$ ls backend/*.json
backend/checkpoints.json     <-- written by the sandbox
backend/todos.json           <-- written by the sandbox
backend/package.json
backend/project-config.json
backend/tsconfig.json
[WebServer] [CheckpointStore] Loaded 0 checkpoints from <repo>/backend/checkpoints.json

Root cause is in backend/src/server.ts, pre-existing on main:

const todoStore = new TodoStore();              // no data dir at all
const checkpointStore = new CheckpointStore(basePath);   // basePath, not dataDir

basePath is only set when Electron passes its userData path. Every CLI, container and E2E instance configures itself with CLAUDIA_DATA_DIR, so basePath is undefined and both stores fall through to their join(__dirname, '..') legacy path.

Consequences: run this suite in your working checkout and it overwrites your live instance's backend/checkpoints.json (149 KB of real checkpoints here); in a container both files are lost on every image update — the exact failure the data-directory work existed to prevent.

Fixed in fix(server): keep checkpoints and todos inside CLAUDIA_DATA_DIR, plus two structural guards in data-dir-wiring.test.ts. Both stores already had working basePath? seams with passing per-store suites — the bug was purely at the call site, which no store-level test can see. The invariant asserted instead is: once dataDir has been resolved from it, basePath must never be read again; and no *Store may be constructed with an empty argument list. Reverting either line fails both tests:

× createApp wiring > uses basePath only to resolve dataDir, never to construct a store
  → expected [ Array(1) ] to deeply equal []
× createApp wiring > constructs no state-writing store with an empty argument list
  → expected [ 'TodoStore' ] to deeply equal []

🔴 2. 00-isolation could never have caught that — it runs first

Two separate reasons it passed while the leak happened on every run:

  1. DEFAULT_STATE_FILES named 3 of the 8 legacy-location files. checkpoints.json and todos.json were not among them.
  2. More fundamentally: it runs first, when the backend has loaded its stores and served one workspace and nothing else. Most state files are written lazily by activity — checkpoints.json on the first task, todos.json on the first TODO — so a leak triggered by later specs is invisible to it no matter how complete the file list is.

Fixed: the list is now exhaustive (with the grep to re-derive it in the comment), and the same assertion is repeated in a new 08-no-state-leaks.spec.ts — last file the runner reaches, after the suite has created workspaces, spawned tasks, taken checkpoints and round-tripped settings. Plus the mirror assertion that the sandbox dir is the one that got written, so an empty leak list cannot mean "nothing was written anywhere".

Proof it now catches the original bug — reverting the server.ts fix and running the full suite:

✘ 28 e2e/tests/08-no-state-leaks.spec.ts:24 › no state escaped the sandbox over the whole run
  Error: a store wrote outside CLAUDIA_DATA_DIR ...
  +   "<repo>/backend/checkpoints.json",
✘ 29 e2e/tests/08-no-state-leaks.spec.ts:34 › the sandbox state dir is the one that actually got written
  Error: checkpoints.json must exist in the sandbox state dir
  2 failed, 27 passed

🔴 3. The suite overwrote frontend/dist — a real production artifact

buildStack() ran vite build into frontend/dist with VITE_CLAUDIA_BACKEND_PORT=4801 compiled in. That path is not scratch space:

  • backend/src/server.ts:7477app.use(express.static(frontendDistPath)), the tunnel/mobile route
  • electron/main.ts:141loadFile(join(__dirname, '..', 'frontend', 'dist', 'index.html')) in a packaged build

So running the suite locally left the developer's tunnel UI — and any Electron build made from that tree — pointing at 127.0.0.1:4801. Silently, because dist/ is gitignored and nothing reports the damage.

Fixed: builds and previews frontend/dist-e2e; frontend/dist is never touched. Verified — after two full runs, frontend/dist does not exist and frontend/dist-e2e does.

🟡 4. CLAUDIA_E2E_* port overrides bought no real isolation

RUN_ROOT was one fixed path regardless of ports, and prepareHarness() rmSyncs it at config load. So the documented way to run two suites at once — different ports — has the second run delete the first run's state dir out from under its live backend. Relevant in a repo whose own orchestrator drives many worktrees in parallel. Fixed: RUN_ROOT is keyed by the backend port (~/.claudia-e2e-4801).

🟡 5. assertSafePort ran after assertPortFree

With CLAUDIA_E2E_BACKEND_PORT=4001 and the dev server up you got "port already in use" rather than the accurate refusal. Fixed — safe-port check now runs first, before any port is probed or any state wiped:

$ CLAUDIA_E2E_BACKEND_PORT=4001 npx playwright test
Error: [e2e] REFUSING TO START: backend port 4001 belongs to the developer's live dev server.
    at assertSafePort (e2e/harness/env.ts:86:15)
    at playwright.config.ts:28:5

$ CLAUDIA_E2E_FRONTEND_PORT=5173 npx playwright test
Error: [e2e] REFUSING TO START: frontend port 5173 belongs to the developer's live dev server.
    at playwright.config.ts:29:5

Nothing was created or probed in either case; ~/.claudia-e2e-4001 never appeared and 4001 stayed healthy.

🟡 6. One genuinely vacuous assertion in 00-isolation

The block commented "Drive a real write through the UI, then prove it landed in the sandbox" opened and closed the Settings modal — which writes nothing — and then asserted:

expect(config, 'sandbox config.json must be readable JSON').toBeTruthy();

JSON.parse of a file the harness itself seeded. It proves nothing about the backend. Fixed: adds a workspace through the UI, asserts the path reaches the active workspace list in the sandbox state file, removes it and asserts it leaves. (Reading the parsed list, not the raw file — a removed workspace is retained under recentWorkspaces, so a substring match would have made the delete half pass vacuously. I wrote that bug first and the test caught it.)


Suite runs

Four full back-to-back runs completed at normal machine load, zero retries, zero flakes:

Unmodified PR (+ #248 merged locally), run 1 → run 2:

  27 passed (29.1s)
  27 passed (27.4s)

Full per-test listing from run 2 — all 27 green:

  ✓   1 00-isolation.spec.ts:21 › the sandboxed ports are not the developer dev-server ports (239ms)
  ✓   2 00-isolation.spec.ts:26 › the backend under test reads and writes only the sandboxed state dir (662ms)
  ✓   3 00-isolation.spec.ts:49 › the running backend is reachable on the sandboxed port only (104ms)
  ✓   4 01-app-loads.spec.ts:12 › shell renders and the WebSocket reaches connected (213ms)
  ✓   5 01-app-loads.spec.ts:23 › boot produces no console errors (204ms)
  ✓   6 01-app-loads.spec.ts:32 › the browser talks to the sandboxed backend, never the dev server (213ms)
  ✓   7 02-workspace-lifecycle.spec.ts:20 › adds a workspace from an absolute path (699ms)
  ✓   8 02-workspace-lifecycle.spec.ts:33 › renames the workspace inline (360ms)
  ✓   9 02-workspace-lifecycle.spec.ts:47 › the workspace and its new name survive a full page reload (249ms)
  ✓  10 02-workspace-lifecycle.spec.ts:58 › removes the workspace (476ms)
  ✓  11 03-task-lifecycle.spec.ts:25 › creates a task and spawns the CLI with the prompt (3.7s)
  ✓  12 03-task-lifecycle.spec.ts:46 › renders the CLI output in the terminal (351ms)
  ✓  13 03-task-lifecycle.spec.ts:57 › delivers follow-up input and renders the new output (475ms)
  ✓  14 03-task-lifecycle.spec.ts:71 › archives the task and it leaves the active list (421ms)
  ✓  15 04-reload-reconnect.spec.ts:27 › a running task survives a full page reload with its history intact (2.8s)
  ✓  16 04-reload-reconnect.spec.ts:55 › the reattached terminal still delivers input to the live process (464ms)
  ✓  17 04-reload-reconnect.spec.ts:72 › history survives repeated reloads without duplicating (410ms)
  ✓  18 05-terminal-resize.spec.ts:55 › a small width change is suppressed; a large one resizes the PTY (4.9s)
  ✓  19 05-terminal-resize.spec.ts:119 › terminal content survives a resize without corruption (492ms)
  ✓  20 05-terminal-resize.spec.ts:145 › output that arrives during the post-resize buffer window still lands (468ms)
  ✓  21 06-settings.spec.ts:41 › a backend-persisted setting survives a reload (1.4s)
  ✓  22 06-settings.spec.ts:67 › the setting can be turned back off and that also persists (1.4s)
  ✓  23 06-settings.spec.ts:86 › theme choice persists across a reload (970ms)
  ✓  24 07-file-explorer.spec.ts:36 › lists the workspace tree (824ms)
  ✓  25 07-file-explorer.spec.ts:46 › expands a directory to reveal its children (481ms)
  ✓  26 07-file-explorer.spec.ts:56 › opens a file and renders its content (470ms)
  ✓  27 07-file-explorer.spec.ts:73 › opens a nested file with the content written by the fixture (615ms)

With my fixes applied, run A → run B:

  27 passed (29.5s)
  27 passed (28.1s)

The specs themselves are solid. No spec was deleted and none needed to be.

Checks that came back clean

Security — the ?backendUrl= seam does not exist. The review brief described a runtime query-param seam in frontend/src/config/api-config.ts; there isn't one. The seam is build-time VITE_CLAUDIA_BACKEND_PORT, and Vite constant-folds it out entirely. Plain vite build with no env set:

prod bundle occurrences of 'backendUrl': 0
prod bundle occurrences of 'VITE_CLAUDIA_BACKEND_PORT': 0
prod bundle occurrences of '4801': 0
prod bundle occurrences of 'CLAUDIA_E2E': 0

folded getBackendPort():  I={BACKEND:4001,FRONTEND:5173,OPENCODE:4097};`${I.BACKEND}`,`${I.BACKEND}`;

The override branch is eliminated at build time — a production bundle cannot be steered by a query param, a runtime env var, or anything else. Nothing to gate. ✅

The bundle-port precondition genuinely works. I rebuilt frontend/dist without the override and re-ran with CLAUDIA_E2E_SKIP_BUILD=1:

[e2e] CLAUDIA_E2E_SKIP_BUILD=1 — reusing existing dist/ output
Error: [e2e] REFUSING TO RUN: the built frontend does not reference port 4801,
       so it would fall back to the developer's backend on 4001.
    at assertBundleTargetsSandbox (e2e/harness/build.ts:51:15)

Teardown on the failure path. Injected a spec that spawns a real fake-claude PTY and then fails hard:

✘ 1 [chromium] › 99-tmp-failsim.spec.ts › FAILSIM: spawns a real fake-claude then fails hard (3.2s)
[e2e] swept leftover fake-claude processes
  1 failed

After it: pgrep -fl 'claudia-e2e|vite preview|dist/index.js' → empty; ~/.claudia-e2e-* gone; 4001 still 200. ✅
(One caveat, not a defect: a run killed with SIGINT skips globalTeardown and leaves the state dir. It is self-healing — prepareHarness() wipes it on the next run.)

State dir is under homedir(), not os.tmpdir()join(homedir(), '.claudia-e2e-<port>'), with the macOS /varvalidateWorkspacePath blocklist rationale documented in place. ✅

Test-quality audit, all 8 spec files. Exactly one waitForTimeout in the suite (05-terminal-resize.spec.ts:97, 1000 ms) and one bare setTimeout(750) inside a settle poll — both assert the absence of an event, which a web-first poll cannot express, and both are documented as such. No "sleep until hopefully ready" anywhere. The loose range assertions (occurrences <= 3 in 04 and 05) are deliberate and justified in comments — an exact === 1 would be wrong because of prompt redraw. The only genuinely vacuous assertion was #6 above. Nothing else needed tightening, and nothing needed deleting. ✅

Frontend source changes are inert. Diffed every touched component against origin/main: data-testid / data-* attributes, one optional panelId prop on CollapsiblePanel, and the getBackendPort() extraction. No behavioural change, no rebase duplication (each testid appears once).

$ cd frontend && npx tsc --noEmit   → exit 0
$ npx vitest run                    → Test Files 21 passed (21)
                                      Tests 1068 passed (1068)
$ npx tsc --noEmit -p tsconfig.e2e.json → exit 0

CI + coverage gating. e2e.yml uses npm ci ✅, npx playwright install --with-deps chromium ✅, and is workflow_dispatch + nightly schedule only — no pull_request trigger, so it cannot block a merge ✅. e2e/ and tsconfig.e2e.json are invisible to the unit suites (every vitest config is package-rooted at src/**) and to the coverage gate (the new-file floor is scoped to (backend|frontend|shared)/src and electron/; vite-env.d.ts is exempt via /\.d\.ts$/).

$ npm run coverage:run
  ✓ ratchet: no repo-wide regression
  ✓ new-file floor: 0 new source file(s) meet the 60% floor
  COVERAGE EXIT=0

The developer's live instance was never touched

Before, between and after every run:

$ curl -s -o /dev/null -w '%{http_code}' http://localhost:4001/api/health
200

$ ls <real data dir>/*.json | diff baseline -   → IDENTICAL FILE SET
$ find ~/.claudia -maxdepth 1 -exec stat -f '%m %N' {} \; | diff baseline -   → IDENTICAL
$ pgrep -fl 'claudia-e2e|vite preview|dist/index.js'   → (empty)

One thing I could not certify, and why

Both machines-under-test runs I attempted late in the review were done while this box was carrying 17 other agent tasks. Under that load individual specs failed on locator.click timeouts with wall clocks of 4–9 minutes against a 60 s budget:

21:11:33 load 51.51
21:15:40 load 63.02      <-- my own load sampler was starved for 4 minutes here
[TaskSpawner] System wake detected {"gapMs":231994,"gapMinutes":4}

The sandboxed backend's own sleep/wake detector firing on a 232-second event-loop stall is unambiguous CPU starvation, not a spec race. The four runs I completed at normal load were all clean, and the failures were in different specs each time with no consistent signature.

I deliberately did not add retries or inflate timeouts to paper over this. retries: 0 is the right call and the PR's reasoning for it is correct. Worth watching in the nightly history: a cold 2-vCPU GitHub runner booting a backend, a vite preview and Chromium may find actionTimeout: 15_000 tight. If the nightly goes red on timeouts rather than assertions, raise the timeouts under process.env.CI — do not add retries.

Commits pushed

test(e2e): make the isolation assertions non-vacuous
test(e2e): stop the harness writing over production build output and state
fix(server): keep checkpoints and todos inside CLAUDIA_DATA_DIR

Plain push, no rebase — 4b3b209 is still the parent. The #248 merge I needed locally is not on the branch; the dependency stands as documented.

@kovtcharov kovtcharov left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correction to my review above — one more real defect, now fixed (5f2fd26)

I marked the test-quality audit clean. That was wrong on one point, and I only caught it because the machine finally went quiet enough for a confirming run of the final state.

05-terminal-resize.spec.ts asserts:

expect(frames.length, 'the large resize must send exactly one task:resize').toBe(settled + 1);

That is over-specified. Changing the viewport moves width and height, and the layout settles in more than one pass — cols reaches its final value immediately while rows changes again a frame later as the surrounding chrome reflows:

frames=[{"cols":87,"rows":35,"t":7674},{"cols":35,"rows":34,"t":19960},{"cols":35,"rows":33,"t":20136}]
                                         ^-- large resize            ^-- rows-only, 176ms later

That third frame is a rows-only delta. The <=2 suppression guard is a column guard — it does not suppress it, and shouldn't. So the frame is legitimate output, not a bug, and the assertion fails on it:

✘ 18 05-terminal-resize.spec.ts:55 › a small width change is suppressed; a large one resizes the PTY
  Error: the large resize must send exactly one task:resize
  Expected: 2   Received: 3

This is not the CPU-starvation noise I described in the review — there is no timeout and no System wake detected marker; it is a plain assertion mismatch. The two frames normally coalesce inside the client's 150 ms debounce window, which is why it passed four full runs in a row. Once the box is loaded enough to skew the layout passes apart they separate. A nightly CI runner is exactly where that skew shows up, so this would have flaked in CI.

Fixed by asserting the property that actually matters — every frame after the wobble belongs to the large resize, i.e. none sits inside the ±2 suppression band around the settled baseline:

const added = frames.slice(settled);
expect(added.length).toBeGreaterThan(0);
for (const frame of added) {
    expect(frame.cols).toBeLessThan(colsBefore - 2);
}

A leaked wobble frame would sit at ~colsBefore and still fail, so the regression the test exists to catch is still caught — it just no longer constrains how many layout passes the client makes.

Verified 3× in separate processes at load ~55 (the condition that exposed it):

--- iteration 1 (load: 54.54) ---  3 passed (28.6s)
--- iteration 2 (load: 60.74) ---  3 passed (29.3s)
--- iteration 3 (load: 54.40) ---  3 passed (19.4s)

Also confirmed in that run: the new 08-no-state-leaks spec works in suite position

✓ 28 08-no-state-leaks.spec.ts:35 › the sandbox state dir is the one that actually gets written (4.2s)
✓ 29 08-no-state-leaks.spec.ts:54 › no state escaped the sandbox over the whole run (577ms)

That closes the one caveat I left open — every spec in the final state has now been observed green, and the only failure across all runs of the fixed branch was the resize assertion above, which is fixed.

Verdict stands: FIXED. One note for whoever runs this next: --repeat-each does not work on this suite (it is deliberately stateful and serial — repeats duplicate task prompts and trip toHaveCount(1)). Re-run whole specs in separate processes instead.

Boots a real backend, a real frontend bundle and a real Chromium against
dedicated ports (4801/5801) with an isolated state dir (~/.claudia-e2e) and
a fake Claude CLI on PATH. 27 specs across 8 flows: isolation, app boot,
workspace lifecycle, task lifecycle, reload/reconnect, terminal resize,
settings round-trip, file explorer.

Isolation is asserted, not assumed. The suite previously contaminated the
developer's live instance on 4001, so there are several independent guards:
  - assertSafePort() refuses to start on 4001/5173
  - assertPortFree() refuses to attach to somebody else's server
  - an in-page fetch/XHR/WebSocket guard throws on a forbidden port
  - the built bundle is checked for the sandboxed port before any test runs
  - CLAUDIA_SHARED_MCP=0 so the sandboxed backend never probes-and-adopts
    the developer's shared Playwright MCP server on 4022
  - 00-isolation.spec asserts the legacy state files and the shared-MCP pid
    file were never created

State lives under homedir(), not os.tmpdir(): on macOS tmpdir resolves under
/var, which validateWorkspacePath blocklists, so workspace creation would be
rejected before reaching the code under test.

Task rows are located by the `title` attribute of task-prompt (the verbatim
prompt), not by rendered text: the backend now assigns a fallback display
title derived from the prompt, so the visible text is not the prompt.

The resize spec's baseline viewport is set inside the project `use`, after
the device spread — devices['Desktop Chrome'] carries its own 1280x720
viewport that silently overrode the top-level one, turning the "10px wobble"
into a 160px resize. The spec now also waits for the resize stream to settle
before wobbling and asserts the wobble adds zero frames and the large resize
adds exactly one.

Frontend source changes are purely additive test hooks — data-testid and
data-* attributes plus an optional panelId prop on CollapsiblePanel — and a
build-time VITE_CLAUDIA_BACKEND_PORT override in api-config.ts. No runtime
behaviour changes.

Depends on the shared ESM .js-extension fix (split into its own PR):
`node backend/dist/index.js` cannot load without it, so the suite does not
run from this branch until that lands.

.gitignore's `node_modules/` (trailing slash) matched directories only; in
a git worktree node_modules is typically a symlink and showed as untracked.

CI runs on workflow_dispatch and a nightly schedule only. It is deliberately
NOT a required PR check on day one; promote it to `pull_request:` once the
nightly history is green.
createApp handed CheckpointStore `basePath` and TodoStore nothing at all.
`basePath` is only set when Electron passes its userData directory, so every
CLI, container and E2E instance — which configure themselves with
CLAUDIA_DATA_DIR — fell through to each store's legacy fallback and wrote
`backend/checkpoints.json` and `backend/todos.json` into the source tree.

That is not a cosmetic misplacement. The files land in the checkout, so a
sandboxed second instance overwrites the checkpoints and todos of the
developer's live one, and a container loses both on every image update — the
exact failure the data-directory work existed to prevent.

Both stores already had a working `basePath?` seam and both have passing
per-store suites; the bug was purely in the call site, which no store-level
test can see. The guard added here is therefore structural: once `dataDir`
has been resolved from it, `basePath` must never be read again, and no store
may be constructed with an empty argument list. Reverting either line of this
fix fails those two tests.
…state

Three sandbox escapes the suite did not cover.

1. The frontend was built into `frontend/dist` with the sandbox backend port
   compiled in. That path is a real production artifact: server.ts serves it
   on the tunnel/mobile route and electron/main.ts loads it in a packaged
   app. Running the suite locally therefore left the developer's tunnel UI —
   and any Electron build made from that tree — pointing at 127.0.0.1:4801,
   silently, since dist/ is gitignored and nothing reports the damage. The
   suite now builds and previews `frontend/dist-e2e` and never touches dist.

2. RUN_ROOT was a single fixed path, so the CLAUDIA_E2E_* port overrides —
   the documented way to run two suites at once, and routine in a repo whose
   own orchestrator drives many worktrees in parallel — bought no isolation
   at all: the second run's prepareHarness() wipes the first run's state dir
   out from under its live backend. It is now keyed by the backend port.

3. assertSafePort ran after assertPortFree, so CLAUDIA_E2E_BACKEND_PORT=4001
   with the dev server up reported "port already in use" instead of naming
   the actual mistake. It now runs first, before any port is probed and
   before any state is wiped.

DEFAULT_STATE_FILES also named only three of the backend's legacy-location
state files. See the following commit for why that mattered.
00-isolation claimed to "drive a real write through the UI, then prove it
landed in the sandbox", but opened and closed the Settings modal — which
writes nothing — and then asserted `expect(JSON.parse(config)).toBeTruthy()`,
which proves only that a file the harness itself seeded is still valid JSON.
It now adds a workspace through the UI, asserts the path reaches the active
workspace list in the sandbox state file, removes it again and asserts it
leaves. (Reading the parsed list, not the raw file: a removed workspace is
retained under `recentWorkspaces`, so a substring match would have made the
delete half pass vacuously.)

The bigger gap was structural. 00-isolation runs FIRST, when the backend has
loaded its stores and served one workspace and nothing else. Most state files
are written lazily, by activity — checkpoints.json on the first task,
todos.json on the first TODO — so a store wired up without a data directory
leaks into the checkout only after this spec has already reported green.
That is exactly what happened: checkpoints.json was written to backend/ on
every single run while the isolation spec passed.

So the assertion is repeated in 08-no-state-leaks, the last file the runner
reaches, after the suite has created workspaces, spawned tasks, taken
checkpoints and round-tripped settings — plus the mirror assertion that the
sandbox state dir is the one that actually got written, so an empty leak list
cannot mean "nothing was written anywhere". DEFAULT_STATE_FILES now names
every legacy-location file rather than a sample of three.
"the large resize must send exactly one task:resize" is over-specified, and
fails for a reason unrelated to the behaviour under test.

Changing the viewport moves width AND height, and the layout settles in more
than one pass: cols reaches its final value immediately while rows can change
again a frame later as the surrounding chrome reflows. That second frame is a
rows-only delta, which the <=2-column suppression guard does not — and should
not — suppress, so it is legitimate output, not a bug:

  frames=[{cols:87,rows:35,t:7674},{cols:35,rows:34,t:19960},{cols:35,rows:33,t:20136}]

The two frames land 176ms apart. They usually coalesce inside the client's
150ms debounce window, which is why this passed run after run; once the box is
loaded enough to skew the layout passes apart they separate, and the exact
count fails. A nightly CI runner is exactly where that skew shows up.

Assert the property that actually matters instead: every frame after the
wobble belongs to the large resize, i.e. none sits inside the +-2 suppression
band around the settled baseline. A leaked wobble frame would sit at
~colsBefore and still fail, so the regression the test exists to catch is
still caught.
#261 made a token mandatory on every /api route and WebSocket upgrade, so
the suite's direct API call (00-isolation) got a 401.

- Direct API calls present the sandbox token via authHeaders(), read with
  the backend's own getAuthToken() against CLAUDIA_DATA_DIR. The harness
  mints it before the backend boots so the two can never race to mint
  different tokens.
- New isolation assertion: no token, or a wrong token, is refused with
  401; the sandbox token is accepted.
- 01-app-loads asserts the browser authenticates through the real
  loopback bootstrap (/api/auth/local, 200), never sees the token gate,
  and that its WebSocket carries ?token=. The suite does not bypass the
  app's own auth path.
- webServer readiness probes /api/health (unauthenticated) instead of
  /api/tasks, which only "passed" because Playwright treats 401 as up.
- auth-token, mcp-token, instance.json and task-histories added to the
  never-created legacy-location list.
- Windows: PATH uses the platform delimiter, and USERPROFILE/APPDATA point
  into the sandbox with a refusing claude.cmd first on PATH, so a Windows
  run cannot write to the real profile or reach the real Claude CLI.
…ists

The isolation spec asserted every legacy backend/ location was ABSENT.
That fails on any checkout where other tooling left a file there, and
says nothing about the sandbox when it does: the backend unit suite does
not set CLAUDIA_DATA_DIR, so mcp-auth mints backend/mcp-token (and the
shared-MCP tests leave .shared-playwright-mcp-*.log) during a normal
`npm test -w backend`. Run unit tests, then the browser suite, and 00-
isolation went red over a file the sandboxed backend never touched.

prepareHarness() now snapshots each legacy location (mtime, or null when
absent) before the backend boots, and the spec asserts the run neither
created nor modified any of them. On a fresh checkout — CI — the baseline
is all-null, so this is exactly the old "never created" check; on a used
checkout it still catches the sandbox writing there.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant