From 6fdfb6cb3ae987114edc8c416a8d45acd61b68f0 Mon Sep 17 00:00:00 2001 From: "Sebastian (Tiedtke) Huckleberry" Date: Tue, 28 Jul 2026 09:36:47 -0700 Subject: [PATCH] docs: document eval concurrency defaults Signed-off-by: Sebastian (Tiedtke) Huckleberry --- docs/eval/index.md | 4 +++- docs/eval/quickstart.md | 2 ++ docs/eval/reference.md | 2 ++ 3 files changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/eval/index.md b/docs/eval/index.md index 8ea11cf3..70ed65d9 100644 --- a/docs/eval/index.md +++ b/docs/eval/index.md @@ -82,7 +82,9 @@ When you run `runme eval` without extra path configuration, Runme uses these def - Dataset path: `./evals/tasks` - Jobs directory: `.runme/evals/jobs` - Agent: `oracle`, Harbor's built-in oracle solution runner -- Environment: Runme's Harbor environment +- Environment: Runme's Harbor environment, with one concurrent trial by default + +When you select another Harbor environment, such as `--env docker`, Runme uses Harbor's concurrency default instead. Override either behavior with passthrough Harbor flags, for example `-- --n-concurrent 4`. ## Commands diff --git a/docs/eval/quickstart.md b/docs/eval/quickstart.md index b209ed2c..3975031b 100644 --- a/docs/eval/quickstart.md +++ b/docs/eval/quickstart.md @@ -80,6 +80,8 @@ To run the same rubric with the deterministic oracle solution runner, replace `c Use `--env` to select one of the Harbor environments provided by the task dataset. When `--env` is omitted, Runme uses its default environment. For Docker environment configuration and task authoring details, see Harbor's [task environment docs](https://www.harborframework.com/docs/tasks#environment). +Runme's default environment runs one concurrent trial because it shares the host workspace. Other Harbor environments, including Docker, use Harbor's configured or default concurrency. Pass Harbor flags after `--` to override concurrency, such as `-- --n-concurrent 4`. + The RewardKit example also provides a Docker environment. Pass `OPENAI_API_KEY` explicitly because Docker runs hermetically and does not reuse your local agent OAuth session: ```sh diff --git a/docs/eval/reference.md b/docs/eval/reference.md index 46d7bb44..5610f4a5 100644 --- a/docs/eval/reference.md +++ b/docs/eval/reference.md @@ -40,6 +40,8 @@ Important flags: - `--runme-harbor-bin`: `runme-harbor` executable. - `--debug`: print delegated commands. +Runme's environment defaults to one concurrent trial. Other Harbor environments, such as `--env docker`, use Harbor's configured or default concurrency. Pass Harbor flags after `--`, for example `-- --n-concurrent 4`, to override concurrency for any environment. + Supported agents: - `oracle`: Harbor's built-in oracle solution runner. It runs the task's provided solution and is not an external agent CLI.