Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion docs/eval/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,9 @@ When you run `runme eval` without extra path configuration, Runme uses these def
- Dataset path: `./evals/tasks`
- Jobs directory: `.runme/evals/jobs`
- Agent: `oracle`, Harbor's built-in oracle solution runner
- Environment: Runme's Harbor environment
- Environment: Runme's Harbor environment, with one concurrent trial by default

When you select another Harbor environment, such as `--env docker`, Runme uses Harbor's concurrency default instead. Override either behavior with passthrough Harbor flags, for example `-- --n-concurrent 4`.

## Commands

Expand Down
2 changes: 2 additions & 0 deletions docs/eval/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,8 @@ To run the same rubric with the deterministic oracle solution runner, replace `c

Use `--env` to select one of the Harbor environments provided by the task dataset. When `--env` is omitted, Runme uses its default environment. For Docker environment configuration and task authoring details, see Harbor's [task environment docs](https://www.harborframework.com/docs/tasks#environment).

Runme's default environment runs one concurrent trial because it shares the host workspace. Other Harbor environments, including Docker, use Harbor's configured or default concurrency. Pass Harbor flags after `--` to override concurrency, such as `-- --n-concurrent 4`.

The RewardKit example also provides a Docker environment. Pass `OPENAI_API_KEY` explicitly because Docker runs hermetically and does not reuse your local agent OAuth session:

```sh
Expand Down
2 changes: 2 additions & 0 deletions docs/eval/reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,8 @@ Important flags:
- `--runme-harbor-bin`: `runme-harbor` executable.
- `--debug`: print delegated commands.

Runme's environment defaults to one concurrent trial. Other Harbor environments, such as `--env docker`, use Harbor's configured or default concurrency. Pass Harbor flags after `--`, for example `-- --n-concurrent 4`, to override concurrency for any environment.

Supported agents:

- `oracle`: Harbor's built-in oracle solution runner. It runs the task's provided solution and is not an external agent CLI.
Expand Down