RFC: Task sampling and in-training evaluation (RFC 013, #1256) #1445

Problem Statement

The Task API lets an environment publish its tasks, but nothing in OpenEnv decides which task a training episode runs, and there is no shared way to evaluate a checkpoint on an environment's test split with fixed seeds. Small models also waste most GRPO groups on tasks they always or never solve. RFC 013 (#1256) proposes both pieces; this issue tracks it and asks three questions about its scope.

Proposed Solution

Trainer-side task sampling and in-training evaluation, with no change to the environment boundary: a TaskSampler protocol with SequentialSampler and UniformSampler in core, a CurriculumSampler that targets GRPO signal offered as an example, a generic TaskAPIClient, an EnvEvalHarness built from the collect pipeline's pieces with compare_eval_results, an EvalSchedule, and openenv eval. Preliminary results from a laptop (the pilot's seed and three more) are in the RFC's Preliminary Results.

Alignment with Principles

Principles Addressed

• Minimize lifecycle deltas (training = production)
• Minimize human-agent divergence
• Be hands-on (working implementations)
• Design for LLMs

Training and evaluation share one interface and one rollout path. The samplers were run before they were proposed; the pilot's code and logs are public, and so are the trial's.

Trade-offs

Impact

Files/Systems Affected

• src/openenv/core/: TaskSampler, SequentialSampler, UniformSampler, TaskAPIClient, EnvEvalHarness, compare_eval_results, EvalSchedule
• src/openenv/core/harness/: the seed passed to the session, a record for failed episodes that keeps apart a reward of None from the environment, a failure of the policy and a failure of the harness or the model endpoint, generated token counts in the episode record, and a model step that turns a text answer into a single-turn environment's tool call
• src/openenv/core/llm_client.py: the provider's usage numbers kept
• the CLI: openenv eval, and tasks passed by openenv collect
• envs/reasoning_gym_env/ and envs/finqa_env/: Task API support, with difficulty tiers for reasoning_gym_env
• docs/source/guides/task-api.md: reserved task spec keys, and guidance on tiers and on train and test overlap
• examples/: the curriculum sampler and the reference experiment

Related RFCs

• RFC 000 (phase three, evals), RFC 004, RFC 005, RFC 006, RFC 008 and its repeatability checks (Restore runtime repeatability #1321), RFC 012
• RFC: GRPO recipes #924 (recipes), docs: add hf rl environment datasets rfc #795 (environment datasets), RFC: Environment Services — brokered, budgeted access to external services (training, GPU jobs, evaluators, simulators) #1382 (environment services) and RFC 014 (RFC 014: Environment Services (design only) #1508), whose posttrain_env (Add experimental posttrain_env prototype (RFC 014) #1500) reports a spend for every episode, scored or not

Implementation Sketch

class TaskSampler(Protocol):
    def sample(self, n: int) -> list[Any]: ...
    def update(self, task: Any, rewards: Sequence[float | None],
               costs: Sequence[float] | None = None) -> None: ...
    def state_dict(self) -> dict[str, Any]: ...
    def load_state_dict(self, state: dict[str, Any]) -> None: ...
    def metrics(self) -> dict[str, float]: ...

Open Questions

1. Should the curriculum sampler ship in core, next to the sequential and uniform samplers, or stay an example?
2. Is a generic client for listing tasks (TaskAPIClient) acceptable in core, given that the Task API guide leaves Task API methods out of the core clients?
3. Should the first version of the evaluation cover Harbor, whose server runs a whole agent per call (run_rollout) and never goes through reset?

Could RFC 013 go on the agenda of the technical committee?

cc @burtenshaw @adithya-s-k @sergiopaniego

This issue was prepared with AI assistance.