test: Playwright browser E2E suite on a fully sandboxed stack - #254
kovtcharov wants to merge 7 commits into
Conversation
kovtcharov
left a comment
There was a problem hiding this comment.
FIXED — two real sandbox escapes found and fixed on the branch; everything else holds up
Independent adversarial review, no prior context. Built and ran the suite against a real backend + real Chromium, with origin/fix/shared-esm-js-extension (#248) merged locally as the PR instructs. Three commits pushed to this branch; the #248 merge is deliberately not included.
The design is genuinely good — assertSafePort, assertPortFree, the in-page fetch/XHR/WebSocket guard, the bundle-port precondition, CLAUDIA_SHARED_MCP=0, and retries: 0 are all the right calls, and I verified each of them actually fires. But the suite was contaminating the checkout it ran in, and its own isolation spec was structurally incapable of noticing.
🔴 1. The sandboxed backend wrote into the developer's checkout on every run
~/.claudia-e2e/... is honoured for most stores — but not all. After every green run:
$ ls backend/*.json
backend/checkpoints.json <-- written by the sandbox
backend/todos.json <-- written by the sandbox
backend/package.json
backend/project-config.json
backend/tsconfig.json
[WebServer] [CheckpointStore] Loaded 0 checkpoints from <repo>/backend/checkpoints.json
Root cause is in backend/src/server.ts, pre-existing on main:
const todoStore = new TodoStore(); // no data dir at all
const checkpointStore = new CheckpointStore(basePath); // basePath, not dataDirbasePath is only set when Electron passes its userData path. Every CLI, container and E2E instance configures itself with CLAUDIA_DATA_DIR, so basePath is undefined and both stores fall through to their join(__dirname, '..') legacy path.
Consequences: run this suite in your working checkout and it overwrites your live instance's backend/checkpoints.json (149 KB of real checkpoints here); in a container both files are lost on every image update — the exact failure the data-directory work existed to prevent.
Fixed in fix(server): keep checkpoints and todos inside CLAUDIA_DATA_DIR, plus two structural guards in data-dir-wiring.test.ts. Both stores already had working basePath? seams with passing per-store suites — the bug was purely at the call site, which no store-level test can see. The invariant asserted instead is: once dataDir has been resolved from it, basePath must never be read again; and no *Store may be constructed with an empty argument list. Reverting either line fails both tests:
× createApp wiring > uses basePath only to resolve dataDir, never to construct a store
→ expected [ Array(1) ] to deeply equal []
× createApp wiring > constructs no state-writing store with an empty argument list
→ expected [ 'TodoStore' ] to deeply equal []
🔴 2. 00-isolation could never have caught that — it runs first
Two separate reasons it passed while the leak happened on every run:
DEFAULT_STATE_FILESnamed 3 of the 8 legacy-location files.checkpoints.jsonandtodos.jsonwere not among them.- More fundamentally: it runs first, when the backend has loaded its stores and served one workspace and nothing else. Most state files are written lazily by activity —
checkpoints.jsonon the first task,todos.jsonon the first TODO — so a leak triggered by later specs is invisible to it no matter how complete the file list is.
Fixed: the list is now exhaustive (with the grep to re-derive it in the comment), and the same assertion is repeated in a new 08-no-state-leaks.spec.ts — last file the runner reaches, after the suite has created workspaces, spawned tasks, taken checkpoints and round-tripped settings. Plus the mirror assertion that the sandbox dir is the one that got written, so an empty leak list cannot mean "nothing was written anywhere".
Proof it now catches the original bug — reverting the server.ts fix and running the full suite:
✘ 28 e2e/tests/08-no-state-leaks.spec.ts:24 › no state escaped the sandbox over the whole run
Error: a store wrote outside CLAUDIA_DATA_DIR ...
+ "<repo>/backend/checkpoints.json",
✘ 29 e2e/tests/08-no-state-leaks.spec.ts:34 › the sandbox state dir is the one that actually got written
Error: checkpoints.json must exist in the sandbox state dir
2 failed, 27 passed
🔴 3. The suite overwrote frontend/dist — a real production artifact
buildStack() ran vite build into frontend/dist with VITE_CLAUDIA_BACKEND_PORT=4801 compiled in. That path is not scratch space:
backend/src/server.ts:7477—app.use(express.static(frontendDistPath)), the tunnel/mobile routeelectron/main.ts:141—loadFile(join(__dirname, '..', 'frontend', 'dist', 'index.html'))in a packaged build
So running the suite locally left the developer's tunnel UI — and any Electron build made from that tree — pointing at 127.0.0.1:4801. Silently, because dist/ is gitignored and nothing reports the damage.
Fixed: builds and previews frontend/dist-e2e; frontend/dist is never touched. Verified — after two full runs, frontend/dist does not exist and frontend/dist-e2e does.
🟡 4. CLAUDIA_E2E_* port overrides bought no real isolation
RUN_ROOT was one fixed path regardless of ports, and prepareHarness() rmSyncs it at config load. So the documented way to run two suites at once — different ports — has the second run delete the first run's state dir out from under its live backend. Relevant in a repo whose own orchestrator drives many worktrees in parallel. Fixed: RUN_ROOT is keyed by the backend port (~/.claudia-e2e-4801).
🟡 5. assertSafePort ran after assertPortFree
With CLAUDIA_E2E_BACKEND_PORT=4001 and the dev server up you got "port already in use" rather than the accurate refusal. Fixed — safe-port check now runs first, before any port is probed or any state wiped:
$ CLAUDIA_E2E_BACKEND_PORT=4001 npx playwright test
Error: [e2e] REFUSING TO START: backend port 4001 belongs to the developer's live dev server.
at assertSafePort (e2e/harness/env.ts:86:15)
at playwright.config.ts:28:5
$ CLAUDIA_E2E_FRONTEND_PORT=5173 npx playwright test
Error: [e2e] REFUSING TO START: frontend port 5173 belongs to the developer's live dev server.
at playwright.config.ts:29:5
Nothing was created or probed in either case; ~/.claudia-e2e-4001 never appeared and 4001 stayed healthy.
🟡 6. One genuinely vacuous assertion in 00-isolation
The block commented "Drive a real write through the UI, then prove it landed in the sandbox" opened and closed the Settings modal — which writes nothing — and then asserted:
expect(config, 'sandbox config.json must be readable JSON').toBeTruthy();JSON.parse of a file the harness itself seeded. It proves nothing about the backend. Fixed: adds a workspace through the UI, asserts the path reaches the active workspace list in the sandbox state file, removes it and asserts it leaves. (Reading the parsed list, not the raw file — a removed workspace is retained under recentWorkspaces, so a substring match would have made the delete half pass vacuously. I wrote that bug first and the test caught it.)
Suite runs
Four full back-to-back runs completed at normal machine load, zero retries, zero flakes:
Unmodified PR (+ #248 merged locally), run 1 → run 2:
27 passed (29.1s)
27 passed (27.4s)
Full per-test listing from run 2 — all 27 green:
✓ 1 00-isolation.spec.ts:21 › the sandboxed ports are not the developer dev-server ports (239ms)
✓ 2 00-isolation.spec.ts:26 › the backend under test reads and writes only the sandboxed state dir (662ms)
✓ 3 00-isolation.spec.ts:49 › the running backend is reachable on the sandboxed port only (104ms)
✓ 4 01-app-loads.spec.ts:12 › shell renders and the WebSocket reaches connected (213ms)
✓ 5 01-app-loads.spec.ts:23 › boot produces no console errors (204ms)
✓ 6 01-app-loads.spec.ts:32 › the browser talks to the sandboxed backend, never the dev server (213ms)
✓ 7 02-workspace-lifecycle.spec.ts:20 › adds a workspace from an absolute path (699ms)
✓ 8 02-workspace-lifecycle.spec.ts:33 › renames the workspace inline (360ms)
✓ 9 02-workspace-lifecycle.spec.ts:47 › the workspace and its new name survive a full page reload (249ms)
✓ 10 02-workspace-lifecycle.spec.ts:58 › removes the workspace (476ms)
✓ 11 03-task-lifecycle.spec.ts:25 › creates a task and spawns the CLI with the prompt (3.7s)
✓ 12 03-task-lifecycle.spec.ts:46 › renders the CLI output in the terminal (351ms)
✓ 13 03-task-lifecycle.spec.ts:57 › delivers follow-up input and renders the new output (475ms)
✓ 14 03-task-lifecycle.spec.ts:71 › archives the task and it leaves the active list (421ms)
✓ 15 04-reload-reconnect.spec.ts:27 › a running task survives a full page reload with its history intact (2.8s)
✓ 16 04-reload-reconnect.spec.ts:55 › the reattached terminal still delivers input to the live process (464ms)
✓ 17 04-reload-reconnect.spec.ts:72 › history survives repeated reloads without duplicating (410ms)
✓ 18 05-terminal-resize.spec.ts:55 › a small width change is suppressed; a large one resizes the PTY (4.9s)
✓ 19 05-terminal-resize.spec.ts:119 › terminal content survives a resize without corruption (492ms)
✓ 20 05-terminal-resize.spec.ts:145 › output that arrives during the post-resize buffer window still lands (468ms)
✓ 21 06-settings.spec.ts:41 › a backend-persisted setting survives a reload (1.4s)
✓ 22 06-settings.spec.ts:67 › the setting can be turned back off and that also persists (1.4s)
✓ 23 06-settings.spec.ts:86 › theme choice persists across a reload (970ms)
✓ 24 07-file-explorer.spec.ts:36 › lists the workspace tree (824ms)
✓ 25 07-file-explorer.spec.ts:46 › expands a directory to reveal its children (481ms)
✓ 26 07-file-explorer.spec.ts:56 › opens a file and renders its content (470ms)
✓ 27 07-file-explorer.spec.ts:73 › opens a nested file with the content written by the fixture (615ms)
With my fixes applied, run A → run B:
27 passed (29.5s)
27 passed (28.1s)
The specs themselves are solid. No spec was deleted and none needed to be.
Checks that came back clean
Security — the ?backendUrl= seam does not exist. The review brief described a runtime query-param seam in frontend/src/config/api-config.ts; there isn't one. The seam is build-time VITE_CLAUDIA_BACKEND_PORT, and Vite constant-folds it out entirely. Plain vite build with no env set:
prod bundle occurrences of 'backendUrl': 0
prod bundle occurrences of 'VITE_CLAUDIA_BACKEND_PORT': 0
prod bundle occurrences of '4801': 0
prod bundle occurrences of 'CLAUDIA_E2E': 0
folded getBackendPort(): I={BACKEND:4001,FRONTEND:5173,OPENCODE:4097};`${I.BACKEND}`,`${I.BACKEND}`;
The override branch is eliminated at build time — a production bundle cannot be steered by a query param, a runtime env var, or anything else. Nothing to gate. ✅
The bundle-port precondition genuinely works. I rebuilt frontend/dist without the override and re-ran with CLAUDIA_E2E_SKIP_BUILD=1:
[e2e] CLAUDIA_E2E_SKIP_BUILD=1 — reusing existing dist/ output
Error: [e2e] REFUSING TO RUN: the built frontend does not reference port 4801,
so it would fall back to the developer's backend on 4001.
at assertBundleTargetsSandbox (e2e/harness/build.ts:51:15)
Teardown on the failure path. Injected a spec that spawns a real fake-claude PTY and then fails hard:
✘ 1 [chromium] › 99-tmp-failsim.spec.ts › FAILSIM: spawns a real fake-claude then fails hard (3.2s)
[e2e] swept leftover fake-claude processes
1 failed
After it: pgrep -fl 'claudia-e2e|vite preview|dist/index.js' → empty; ~/.claudia-e2e-* gone; 4001 still 200. ✅
(One caveat, not a defect: a run killed with SIGINT skips globalTeardown and leaves the state dir. It is self-healing — prepareHarness() wipes it on the next run.)
State dir is under homedir(), not os.tmpdir() — join(homedir(), '.claudia-e2e-<port>'), with the macOS /var → validateWorkspacePath blocklist rationale documented in place. ✅
Test-quality audit, all 8 spec files. Exactly one waitForTimeout in the suite (05-terminal-resize.spec.ts:97, 1000 ms) and one bare setTimeout(750) inside a settle poll — both assert the absence of an event, which a web-first poll cannot express, and both are documented as such. No "sleep until hopefully ready" anywhere. The loose range assertions (occurrences <= 3 in 04 and 05) are deliberate and justified in comments — an exact === 1 would be wrong because of prompt redraw. The only genuinely vacuous assertion was #6 above. Nothing else needed tightening, and nothing needed deleting. ✅
Frontend source changes are inert. Diffed every touched component against origin/main: data-testid / data-* attributes, one optional panelId prop on CollapsiblePanel, and the getBackendPort() extraction. No behavioural change, no rebase duplication (each testid appears once).
$ cd frontend && npx tsc --noEmit → exit 0
$ npx vitest run → Test Files 21 passed (21)
Tests 1068 passed (1068)
$ npx tsc --noEmit -p tsconfig.e2e.json → exit 0
CI + coverage gating. e2e.yml uses npm ci ✅, npx playwright install --with-deps chromium ✅, and is workflow_dispatch + nightly schedule only — no pull_request trigger, so it cannot block a merge ✅. e2e/ and tsconfig.e2e.json are invisible to the unit suites (every vitest config is package-rooted at src/**) and to the coverage gate (the new-file floor is scoped to (backend|frontend|shared)/src and electron/; vite-env.d.ts is exempt via /\.d\.ts$/).
$ npm run coverage:run
✓ ratchet: no repo-wide regression
✓ new-file floor: 0 new source file(s) meet the 60% floor
COVERAGE EXIT=0
The developer's live instance was never touched
Before, between and after every run:
$ curl -s -o /dev/null -w '%{http_code}' http://localhost:4001/api/health
200
$ ls <real data dir>/*.json | diff baseline - → IDENTICAL FILE SET
$ find ~/.claudia -maxdepth 1 -exec stat -f '%m %N' {} \; | diff baseline - → IDENTICAL
$ pgrep -fl 'claudia-e2e|vite preview|dist/index.js' → (empty)
One thing I could not certify, and why
Both machines-under-test runs I attempted late in the review were done while this box was carrying 17 other agent tasks. Under that load individual specs failed on locator.click timeouts with wall clocks of 4–9 minutes against a 60 s budget:
21:11:33 load 51.51
21:15:40 load 63.02 <-- my own load sampler was starved for 4 minutes here
[TaskSpawner] System wake detected {"gapMs":231994,"gapMinutes":4}
The sandboxed backend's own sleep/wake detector firing on a 232-second event-loop stall is unambiguous CPU starvation, not a spec race. The four runs I completed at normal load were all clean, and the failures were in different specs each time with no consistent signature.
I deliberately did not add retries or inflate timeouts to paper over this. retries: 0 is the right call and the PR's reasoning for it is correct. Worth watching in the nightly history: a cold 2-vCPU GitHub runner booting a backend, a vite preview and Chromium may find actionTimeout: 15_000 tight. If the nightly goes red on timeouts rather than assertions, raise the timeouts under process.env.CI — do not add retries.
Commits pushed
test(e2e): make the isolation assertions non-vacuous
test(e2e): stop the harness writing over production build output and state
fix(server): keep checkpoints and todos inside CLAUDIA_DATA_DIR
Plain push, no rebase — 4b3b209 is still the parent. The #248 merge I needed locally is not on the branch; the dependency stands as documented.
kovtcharov
left a comment
There was a problem hiding this comment.
Correction to my review above — one more real defect, now fixed (5f2fd26)
I marked the test-quality audit clean. That was wrong on one point, and I only caught it because the machine finally went quiet enough for a confirming run of the final state.
05-terminal-resize.spec.ts asserts:
expect(frames.length, 'the large resize must send exactly one task:resize').toBe(settled + 1);That is over-specified. Changing the viewport moves width and height, and the layout settles in more than one pass — cols reaches its final value immediately while rows changes again a frame later as the surrounding chrome reflows:
frames=[{"cols":87,"rows":35,"t":7674},{"cols":35,"rows":34,"t":19960},{"cols":35,"rows":33,"t":20136}]
^-- large resize ^-- rows-only, 176ms later
That third frame is a rows-only delta. The <=2 suppression guard is a column guard — it does not suppress it, and shouldn't. So the frame is legitimate output, not a bug, and the assertion fails on it:
✘ 18 05-terminal-resize.spec.ts:55 › a small width change is suppressed; a large one resizes the PTY
Error: the large resize must send exactly one task:resize
Expected: 2 Received: 3
This is not the CPU-starvation noise I described in the review — there is no timeout and no System wake detected marker; it is a plain assertion mismatch. The two frames normally coalesce inside the client's 150 ms debounce window, which is why it passed four full runs in a row. Once the box is loaded enough to skew the layout passes apart they separate. A nightly CI runner is exactly where that skew shows up, so this would have flaked in CI.
Fixed by asserting the property that actually matters — every frame after the wobble belongs to the large resize, i.e. none sits inside the ±2 suppression band around the settled baseline:
const added = frames.slice(settled);
expect(added.length).toBeGreaterThan(0);
for (const frame of added) {
expect(frame.cols).toBeLessThan(colsBefore - 2);
}A leaked wobble frame would sit at ~colsBefore and still fail, so the regression the test exists to catch is still caught — it just no longer constrains how many layout passes the client makes.
Verified 3× in separate processes at load ~55 (the condition that exposed it):
--- iteration 1 (load: 54.54) --- 3 passed (28.6s)
--- iteration 2 (load: 60.74) --- 3 passed (29.3s)
--- iteration 3 (load: 54.40) --- 3 passed (19.4s)
Also confirmed in that run: the new 08-no-state-leaks spec works in suite position
✓ 28 08-no-state-leaks.spec.ts:35 › the sandbox state dir is the one that actually gets written (4.2s)
✓ 29 08-no-state-leaks.spec.ts:54 › no state escaped the sandbox over the whole run (577ms)
That closes the one caveat I left open — every spec in the final state has now been observed green, and the only failure across all runs of the fixed branch was the resize assertion above, which is fixed.
Verdict stands: FIXED. One note for whoever runs this next: --repeat-each does not work on this suite (it is deliberately stateful and serial — repeats duplicate task prompts and trip toHaveCount(1)). Re-run whole specs in separate processes instead.
5f2fd26 to
4bc047d
Compare
4bc047d to
7499976
Compare
Boots a real backend, a real frontend bundle and a real Chromium against
dedicated ports (4801/5801) with an isolated state dir (~/.claudia-e2e) and
a fake Claude CLI on PATH. 27 specs across 8 flows: isolation, app boot,
workspace lifecycle, task lifecycle, reload/reconnect, terminal resize,
settings round-trip, file explorer.
Isolation is asserted, not assumed. The suite previously contaminated the
developer's live instance on 4001, so there are several independent guards:
- assertSafePort() refuses to start on 4001/5173
- assertPortFree() refuses to attach to somebody else's server
- an in-page fetch/XHR/WebSocket guard throws on a forbidden port
- the built bundle is checked for the sandboxed port before any test runs
- CLAUDIA_SHARED_MCP=0 so the sandboxed backend never probes-and-adopts
the developer's shared Playwright MCP server on 4022
- 00-isolation.spec asserts the legacy state files and the shared-MCP pid
file were never created
State lives under homedir(), not os.tmpdir(): on macOS tmpdir resolves under
/var, which validateWorkspacePath blocklists, so workspace creation would be
rejected before reaching the code under test.
Task rows are located by the `title` attribute of task-prompt (the verbatim
prompt), not by rendered text: the backend now assigns a fallback display
title derived from the prompt, so the visible text is not the prompt.
The resize spec's baseline viewport is set inside the project `use`, after
the device spread — devices['Desktop Chrome'] carries its own 1280x720
viewport that silently overrode the top-level one, turning the "10px wobble"
into a 160px resize. The spec now also waits for the resize stream to settle
before wobbling and asserts the wobble adds zero frames and the large resize
adds exactly one.
Frontend source changes are purely additive test hooks — data-testid and
data-* attributes plus an optional panelId prop on CollapsiblePanel — and a
build-time VITE_CLAUDIA_BACKEND_PORT override in api-config.ts. No runtime
behaviour changes.
Depends on the shared ESM .js-extension fix (split into its own PR):
`node backend/dist/index.js` cannot load without it, so the suite does not
run from this branch until that lands.
.gitignore's `node_modules/` (trailing slash) matched directories only; in
a git worktree node_modules is typically a symlink and showed as untracked.
CI runs on workflow_dispatch and a nightly schedule only. It is deliberately
NOT a required PR check on day one; promote it to `pull_request:` once the
nightly history is green.
createApp handed CheckpointStore `basePath` and TodoStore nothing at all. `basePath` is only set when Electron passes its userData directory, so every CLI, container and E2E instance — which configure themselves with CLAUDIA_DATA_DIR — fell through to each store's legacy fallback and wrote `backend/checkpoints.json` and `backend/todos.json` into the source tree. That is not a cosmetic misplacement. The files land in the checkout, so a sandboxed second instance overwrites the checkpoints and todos of the developer's live one, and a container loses both on every image update — the exact failure the data-directory work existed to prevent. Both stores already had a working `basePath?` seam and both have passing per-store suites; the bug was purely in the call site, which no store-level test can see. The guard added here is therefore structural: once `dataDir` has been resolved from it, `basePath` must never be read again, and no store may be constructed with an empty argument list. Reverting either line of this fix fails those two tests.
…state Three sandbox escapes the suite did not cover. 1. The frontend was built into `frontend/dist` with the sandbox backend port compiled in. That path is a real production artifact: server.ts serves it on the tunnel/mobile route and electron/main.ts loads it in a packaged app. Running the suite locally therefore left the developer's tunnel UI — and any Electron build made from that tree — pointing at 127.0.0.1:4801, silently, since dist/ is gitignored and nothing reports the damage. The suite now builds and previews `frontend/dist-e2e` and never touches dist. 2. RUN_ROOT was a single fixed path, so the CLAUDIA_E2E_* port overrides — the documented way to run two suites at once, and routine in a repo whose own orchestrator drives many worktrees in parallel — bought no isolation at all: the second run's prepareHarness() wipes the first run's state dir out from under its live backend. It is now keyed by the backend port. 3. assertSafePort ran after assertPortFree, so CLAUDIA_E2E_BACKEND_PORT=4001 with the dev server up reported "port already in use" instead of naming the actual mistake. It now runs first, before any port is probed and before any state is wiped. DEFAULT_STATE_FILES also named only three of the backend's legacy-location state files. See the following commit for why that mattered.
00-isolation claimed to "drive a real write through the UI, then prove it landed in the sandbox", but opened and closed the Settings modal — which writes nothing — and then asserted `expect(JSON.parse(config)).toBeTruthy()`, which proves only that a file the harness itself seeded is still valid JSON. It now adds a workspace through the UI, asserts the path reaches the active workspace list in the sandbox state file, removes it again and asserts it leaves. (Reading the parsed list, not the raw file: a removed workspace is retained under `recentWorkspaces`, so a substring match would have made the delete half pass vacuously.) The bigger gap was structural. 00-isolation runs FIRST, when the backend has loaded its stores and served one workspace and nothing else. Most state files are written lazily, by activity — checkpoints.json on the first task, todos.json on the first TODO — so a store wired up without a data directory leaks into the checkout only after this spec has already reported green. That is exactly what happened: checkpoints.json was written to backend/ on every single run while the isolation spec passed. So the assertion is repeated in 08-no-state-leaks, the last file the runner reaches, after the suite has created workspaces, spawned tasks, taken checkpoints and round-tripped settings — plus the mirror assertion that the sandbox state dir is the one that actually got written, so an empty leak list cannot mean "nothing was written anywhere". DEFAULT_STATE_FILES now names every legacy-location file rather than a sample of three.
"the large resize must send exactly one task:resize" is over-specified, and
fails for a reason unrelated to the behaviour under test.
Changing the viewport moves width AND height, and the layout settles in more
than one pass: cols reaches its final value immediately while rows can change
again a frame later as the surrounding chrome reflows. That second frame is a
rows-only delta, which the <=2-column suppression guard does not — and should
not — suppress, so it is legitimate output, not a bug:
frames=[{cols:87,rows:35,t:7674},{cols:35,rows:34,t:19960},{cols:35,rows:33,t:20136}]
The two frames land 176ms apart. They usually coalesce inside the client's
150ms debounce window, which is why this passed run after run; once the box is
loaded enough to skew the layout passes apart they separate, and the exact
count fails. A nightly CI runner is exactly where that skew shows up.
Assert the property that actually matters instead: every frame after the
wobble belongs to the large resize, i.e. none sits inside the +-2 suppression
band around the settled baseline. A leaked wobble frame would sit at
~colsBefore and still fail, so the regression the test exists to catch is
still caught.
#261 made a token mandatory on every /api route and WebSocket upgrade, so the suite's direct API call (00-isolation) got a 401. - Direct API calls present the sandbox token via authHeaders(), read with the backend's own getAuthToken() against CLAUDIA_DATA_DIR. The harness mints it before the backend boots so the two can never race to mint different tokens. - New isolation assertion: no token, or a wrong token, is refused with 401; the sandbox token is accepted. - 01-app-loads asserts the browser authenticates through the real loopback bootstrap (/api/auth/local, 200), never sees the token gate, and that its WebSocket carries ?token=. The suite does not bypass the app's own auth path. - webServer readiness probes /api/health (unauthenticated) instead of /api/tasks, which only "passed" because Playwright treats 401 as up. - auth-token, mcp-token, instance.json and task-histories added to the never-created legacy-location list. - Windows: PATH uses the platform delimiter, and USERPROFILE/APPDATA point into the sandbox with a refusing claude.cmd first on PATH, so a Windows run cannot write to the real profile or reach the real Claude CLI.
…ists The isolation spec asserted every legacy backend/ location was ABSENT. That fails on any checkout where other tooling left a file there, and says nothing about the sandbox when it does: the backend unit suite does not set CLAUDIA_DATA_DIR, so mcp-auth mints backend/mcp-token (and the shared-MCP tests leave .shared-playwright-mcp-*.log) during a normal `npm test -w backend`. Run unit tests, then the browser suite, and 00- isolation went red over a file the sandboxed backend never touched. prepareHarness() now snapshots each legacy location (mtime, or null when absent) before the backend boots, and the spec asserts the run neither created nor modified any of them. On a fresh checkout — CI — the baseline is all-null, so this is exactly the old "never created" check; on a used checkout it still catches the sandbox writing there.
7499976 to
f91e5d2
Compare
What
A browser-level end-to-end suite: real built backend + real built frontend bundle + real Chromium, driven by Playwright, against a fully sandboxed Claudia stack with a fake
claudeCLI onPATH. 29 specs, zero retries, one worker.Re-verified under always-on auth (rebased onto
8ce9aa9)Since this was last green, main landed #261 (a token on every
/apiroute and every WebSocket upgrade), plus #259, #262, #258, #249, #255, #272, #252 and #251. This branch is rebased onto that and re-verified:frontend/src/config/api-config.tswas rewritten by feat(auth): require a token on every API route and WebSocket upgrade #261 for auth; this PR edits it forVITE_CLAUDIA_BACKEND_PORT. The resolution keeps both:getBackendPort()drives the web HTTP and WS URLs, and the WS URL still goes through feat(auth): require a token on every API route and WebSocket upgrade #261'swithWsToken().TodoStore(dataDir)/CheckpointStore(dataDir)fix as this PR'sfix(server)commit.server.tskeeps main's (identical) code plus this PR's explanatory comment.data-dir-wiring.test.tskeeps both test sets: fix(data-dir): honor CLAUDIA_DATA_DIR for todos, checkpoints, and shared-MCP pid files #251's behavioural tests (bootcreateApp, prove the files land in the data dir) and this PR's structural guards (nobasePathread afterensureDataDir, no argument-lessnew XStore()), which catch the next store wired up wrong. 10/10 pass.authHeaders()ine2e/harness/env.tsreads the token with the backend's owngetAuthToken(STATE_DIR)(backend/src/auth-token.ts) against the sandbox'sCLAUDIA_DATA_DIR. The harness mints it before the backend boots, so the two processes can never race to mint different tokens./api/auth/local), exactly as a user who opens a browser on the same machine would.00-isolation: an unauthenticated request, and one with a well-formed but wrong token, get 401; the sandbox token gets 200, andauth-tokenlives in the sandbox state dir.01-app-loads:/api/auth/localanswered 200 from the sandboxed backend, the token gate never appears, and every WebSocket URL carries?token=.webServernow waits on/api/health(unauthenticated)./api/tasksreturns 401 now; Playwright counts a 401 as "up", so the old probe was passing by accident of that rule.auth-token,mcp-token,instance.jsonandtask-historiesadded to the legacybackend/locations the sandbox must not write.CLAUDIA_DATA_DIR, so a normalnpm test -w backendmintsbackend/mcp-token.prepareHarness()now snapshots each location (mtime, or null) before the backend boots, and the spec asserts the run neither created nor modified any of them. On a fresh checkout (CI) the baseline is all-null, so this is exactly the old "never created" check.HOMEdoes not relocateos.homedir()andresolveClaudeSpawn()finds the CLI throughAPPDATA/USERPROFILE, notPATH. The harness now points both into the sandbox and puts a refusingclaude.cmdfirst onPATH, and uses the platformPATHdelimiter. A Windows run can therefore neither write into the real profile nor reach the real Claude CLI. The fake CLI itself is a bash script, so the task specs still need a POSIX host.Evidence
Local (Windows 11, the specs that do not spawn the fake CLI), zero retries:
03-task-lifecycle,04-reload-reconnectand05-terminal-resize(10 specs) drive the bash fake CLI and were not run locally: no POSIX host was available on the verifying machine.e2e.ymlruns onworkflow_dispatch+ nightly only, and both require the workflow to be onmain, so those 10 specs are first exercised in CI after merge unless apull_requesttrigger is added.Gates on the same tree:
Coverage (8 spec areas)
isolationbackend/*state, token and lock files and the shared-MCP pid file were never created; backend answers on the sandboxed port;/apirefuses missing/wrong tokens with 401app-loadsconnected, zero console/page errors on boot, browser authenticates via the loopback bootstrap with a token-bearing WS, browser never contacts 4001/5173workspace-lifecyclewindow.confirm) → stays gone after reloadtask-lifecyclereload-reconnectterminal-resizetask:resize; a large resize sends exactly one; content survives resize; output during the 250ms post-resize buffer still landssettingsPUT /api/configand survives reload (both directions); theme persists via localStorage bootstrapfile-explorerIsolation guarantees — and how each is self-asserted
The suite once contaminated the developer's live instance on 4001. Every guard below is checked at runtime, not assumed:
assertSafePort()hard-fails on 4001/5173;assertPortFree()refuses to attach to an existing listener on 4801/5801.assertBundleTargetsSandbox) — a bundle built withoutVITE_CLAUDIA_BACKEND_PORTsilently falls back to 4001.fetch,XMLHttpRequest.openandWebSocketthrow on:4001/:5173; a per-test fixture also asserts every WebSocket targeted the sandboxed backend.CLAUDIA_DATA_DIR+ fakeHOME(and, on Windows,USERPROFILE/APPDATA) under~/.claudia-e2e-<port>(underhomedir(), notos.tmpdir()— on macOS/tmpresolves under/var, whichvalidateWorkspacePathblocklists).claudereachable is the fake fixture; teardownpkills only processes whose cmdline contains the sandbox's own fake-CLI path.CLAUDIA_SHARED_MCP=0so the sandboxed backend never probes-and-adopts the developer's shared Playwright MCP server on 4022.Fixed-port caveat
Ports are fixed at 4801/5801, overridable via
CLAUDIA_E2E_BACKEND_PORT/CLAUDIA_E2E_FRONTEND_PORT, and guarded three ways (forbidden-list, free-check, in-page guard). The state root is keyed by the backend port, so two suites on different ports do not share state.Deliberate bounded wait
There is exactly one
waitForTimeoutin the suite (05-terminal-resize): the property under test is the absence of atask:resizeframe after a 10px wobble, and absence can only be observed by outlasting the 150ms debounce window. Every other wait is a web-first locator/expect.poll.Product-code changes
Purely additive:
data-testid/data-*attributes across components, an optionalpanelIdprop onCollapsiblePanel,data-ws-connectedon the app root, a build-timeVITE_CLAUDIA_BACKEND_PORToverride inapi-config.ts(no runtime/query-param seam), and checkpoints/todos kept insideCLAUDIA_DATA_DIR..gitignore:node_modules/→node_modules(in a worktree it is a symlink, which the trailing-slash form did not match).CI
.github/workflows/e2e.yml:workflow_dispatch+ nightly schedule only,npm ci,npx playwright install --with-deps chromium, artifacts on failure. Not a required check and does not run on pull requests; promote topull_request:once nightly history is green.Run locally