evals: add Codex Agent Runme harness

evals/.gitignore

@@ -0,0 +1,2 @@
+__pycache__/
+*.py[cod]

evals/README.md

@@ -0,0 +1,64 @@
+# Codex Agent evals for Runme Web
+
+This directory contains an external eval harness for Runme Web. It launches a Codex Agent build, creates a task without sending a throwaway turn, prepares one or more deterministic browser tabs through the leased-tab eval API, submits the actual prompt, and verifies the resulting task, tool trace, and browser state.
+
+`cases.json` expands to 116 independently reported trials:
+
+- three baseline cases ported from openai/openai#1087983;
+- 100 isolated write trials covering ten prompt phrasings for redundant-confirmation measurement;
+- direct-URI cases that reject unnecessary Drive search, including a URI read from a notebook cell;
+- notebook display cases for already-open, not-open, and background-tab states;
+- runner-enumeration cases that require the supported `runmeRunners` API; and
+- kernel-selection cases that distinguish `appkernel-js-sandbox` from `appkernel-js`.
+
+Entries may set `repeat` and use `{trial}` or `{trial_token}` placeholders. Expansion happens during case loading, so every trial has a unique ID and a unique page/answer marker while the source file stays reviewable. `--list-cases` prints the expanded dataset and category counts without requiring credentials or launching Codex.
+
+The repository does not import OpenAI monorepo Python packages, bundle internal marketplace plugins, or contain OpenAI credentials. The cases use the Agent build's bundled Browser plugin and Runme's site-provided WebMCP tools. The Codex Apps checkout and Google service-account file are explicit runtime inputs.
+
+Every launched Agent uses a fresh runtime directory with an isolated `HOME`, `CODEX_HOME`, XDG config/cache/data/state directories, npm cache, global Git config path, and Chromium `user-data` profile. Selected cases in one invocation share that isolated Agent process and profile, while each case gets a separate empty task, browser tab set, and Drive copy when needed. The harness copies only the explicitly selected Codex `auth.json` into that temporary `CODEX_HOME`; it does not reuse the caller's Codex configuration or browser profile. Immutable DotSlash and Codex Apps toolchain-artifact caches may be shared so the dev app can boot without copying caller credentials or redownloading build dependencies. The default auth source is the caller's `~/.codex/auth.json`, so the isolated profile uses the same account identity. Pass `--codex-auth-file /path/to/eval-only/auth.json` as well when the eval must use a separate account. Pass `--keep-runtime` only when the isolated logs and non-browser artifacts are needed for debugging. The Chromium profile is always deleted after the Agent exits because it temporarily contains the Drive credential used by the eval.
+
+## Setup
+
+Create a virtual environment and install the small driver dependency set:
+
+```bash
+python3 -m venv .venv
+.venv/bin/pip install -r evals/requirements.txt
+```
+
+The service account must be able to read the source notebooks and create/delete a copy for the write case. Codex authentication defaults to `~/.codex/auth.json` and is copied into an isolated temporary Codex home for each run.
+
+## Run
+
+From the `runmedev/web` repository root:
+
+```bash
+.venv/bin/python evals/run.py \
+  --codex-apps-root /path/to/openai/codex/codex-apps \
+  --service-account-file /path/to/service-account.json
+```
+
+Run one case with `--case google-drive-first-cell`. Add `--keep-runtime` to retain the isolated Codex logs and profile after the run.
+
+Case selectors also match expanded trial prefixes. For example, this runs all ten trials generated by one confirmation template:
+
+```bash
+.venv/bin/python evals/run.py \
+  --codex-apps-root /path/to/openai/codex/codex-apps \
+  --service-account-file /path/to/service-account.json \
+  --case confirmation-add-markdown
+```
+
+Use `--category redundant-confirmation` to run the full 100-trial statistical cohort. `--results-file results.json` atomically checkpoints the aggregate report after every case, including completed/remaining counts, the observed redundant-confirmation rate, and its 95% Wilson interval. If a long run is interrupted, rerun the same selection with the same results file and `--resume`; completed case IDs are validated and skipped, while setup failures are removed from the checkpoint and retried. Each case also retries transient Runme notebook-import failures twice by default after the three in-page import attempts; configure that with `--setup-retries`. The harness continues after individual case failures by default and reports counts by category and failure mode; pass `--fail-fast` only for focused debugging.
+
+The harness exits nonzero if any case fails. Assertions cover setup, prepared-tab claims, unexpected tabs, task completion, required or forbidden answer text, required or forbidden WebMCP trace evidence, visible page state, and persisted Drive notebook state. Write cases confirm the expected cell reached the Drive-backed notebook. Kernel-selection cases additionally verify the persisted cell's `runme.dev/runnerName`, completed execution state, zero exit code, and decoded output. A response that asks for approval instead of performing an authorized write is classified as `redundant_confirmation`.
+
+## Agent build command
+
+The harness starts Codex from the supplied checkout with:
+
+```bash
+pnpm run app --flavor agent --playwright ...
+```
+
+The remaining arguments point at isolated Codex home, SQLite, and browser-profile directories created for the run. The harness uses the checkout's normal dev-app metadata path and refuses to launch if that worktree already has a live Codex app. Pass `--attach-cdp-url` to exercise the cases against an Agent build that is already running instead.