[Matheus-F-Scatolin]

Summary

Adds RFC 013, which proposes trainer-side task sampling and in-training evaluation on top of the existing Task API: a TaskSampler protocol with sequential and uniform samplers in core, a curriculum sampler offered as an example, a generic TaskAPIClient, an EnvEvalHarness built from the openenv collect pipeline's pieces, an EvalSchedule, and an openenv eval command. It targets the "simple evals that can be scored by a script" step of RFC 000's phase three. This PR only adds the RFC and its index entry; no code changes.

The RFC includes results measured on a laptop (Llama-3.2-1B, GRPO + LoRA in MLX, reasoning_gym_env chain_sum in five difficulty tiers): a single-seed pilot, then a trial on three more seeds, with predictions and the rules for reading the results written down before it ran. A curriculum that targets GRPO signal learned more than uniform sampling per environment episode on all four seeds, and per generated token on the three new ones (+5.5, +18.3 and +14.4 points); on the pilot's seed, uniform sampling was ahead per token by 7.7 points. Three new seeds are a pattern, not a significance test. Weighting the curriculum by token cost with the formula of the first version did worse than the signal-only curriculum, and a corrected form matched it per generated token without beating it, so the CurriculumSampler weights by signal only and the samplers proposed here report signal per episode and per unit of the cost the trainer reports.

Pilot code and logs: Matheus-F-Scatolin/OpenEnv@rfc-013-pilot. Trial code and logs: Matheus-F-Scatolin/OpenEnv@rfc-013-trial.

Questions for the maintainers are in #1445.

Changes since 2026-09-27

• 2026-10-12: removed a CLI limitation fixed by fix: LLM endpoint full base URLs (from #1189) #1237, updated the RFC 008 reference to Restore runtime repeatability #1321, and corrected four small inaccuracies.
• 2026-10-16: added results on three more seeds, replaced the token-weighted formula, scoped the first evaluation to environments whose turns the trainer drives, listed the gaps in the collect pipeline, offered the curriculum as an example, linked the trial's code and logs, and stated how the token weights treat unseen keys and many keys.
• 2026-10-22: after review, removed weighting by cost and the cost argument from the curriculum, stated the two hooks a sampler needs in a trainer and their limits in TRL, added metrics() to the protocol, left shares across sources to the trainer, reported how far the signal formula over-predicts, cited prior work on rollout allocation and filtering, narrowed the reference experiment to uniform sampling against the curriculum on five new seeds, answered open question 6 and proposed a direction for open question 5.
• 2026-10-28: removed the prime-rl wrapper sketch and the TRL dataset path, as the approving review suggested, changed how samplers treat a rollout the environment could not score (a reward of None), a rule RFC 014's posttrain_env relies on, so that it counts in groups, episodes and cost but in nothing estimated from rewards, added open question 10, and stated how evaluation counts each case: such a rollout, and one lost to a failure of the infrastructure, leaves success_rate, mean_reward and pass@k and is reported separately, a failure of the policy gets the environment's own score or counts as not solved, mean_reward is read only together with error_rate, and checkpoint comparison lists the tasks it leaves out.

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation
• New environment
• Refactoring

Alignment Checklist

Before submitting, verify:

• I have read the project principles (PRINCIPLES.md) and this PR aligns with our principles
• I have checked the invariants (INVARIANTS.md) and no invariants are violated (one tension is flagged below)
• I have run the pre-submit checks (lint.sh and tests) and addressed all issues. Not run: the PR changes only Markdown in rfcs/, which lint.sh and the tests do not cover, so it adds no new finding.

RFC Status

• Not required (bug fix, docs, minor refactoring)
• RFC exists: this PR is the RFC; tracking issue RFC: Task sampling and in-training evaluation (RFC 013, #1256) #1445
• RFC needed (will create before merge)

Test Plan

Docs only. To rerun the pilot (Apple Silicon, MLX needs the Metal GPU):

git clone --single-branch -b rfc-013-pilot https://github.com/Matheus-F-Scatolin/OpenEnv.git rfc-013-pilot
cd rfc-013-pilot/pilot && uv sync
caffeinate -i uv run python smoke.py train --model llama3.2-1b --sampler curriculum --steps 100000 --max-minutes 150 --eval-every 100 --eval-at-start --eval-per-tier 100 --out runs/pilot
caffeinate -i uv run python smoke.py train --model llama3.2-1b --sampler uniform --steps 100000 --max-minutes 150 --eval-every 100 --eval-at-start --eval-per-tier 100 --out runs/pilot
uv run --no-project --with matplotlib python rfc_figure.py runs/pilot ../figures/pilot-episodes-vs-tokens.png

The pilot's script writes today's date into every prompt, through the model's chat template, and stops after 150 minutes of training, so a rerun on another day or another machine gives other numbers than the stored logs. The trial's trainer fixes the date and stops on generated tokens; run again on seed 0, it reproduced the pilot's stored logs step for step for the first 370 steps of uniform sampling and 158 of the curriculum. To regenerate every table and figure of the trial from its stored logs, without a GPU:

git clone --single-branch -b rfc-013-trial https://github.com/Matheus-F-Scatolin/OpenEnv.git rfc-013-trial
cd rfc-013-trial && uv sync --locked
uv run python -m analysis.report

The README of that branch gives the commands that ran the trial itself.

Alignment Review

Alignment review done by hand against this diff, following the project's alignment-review checklist.

Automated Checks

• Lint: N/A - not run; the PR changes only Markdown in rfcs/
• Debug code: N/A - not run; no code changed

Open RFCs Context

• RFC 000 (project phases): this RFC proposes the first step of its phase three, "simple evals that can be scored by a script".
• RFC 005 (agentic harnesses) and the collect pipeline: the evaluation is built from its pieces; see the flag "Evaluation follows the collect pipeline" below.
• RFC 006 (agentic RL harness interception): its token records are one source of the per-rollout costs a trainer passes to update.
• RFC 008 and Restore runtime repeatability #1321 (runtime repeatability): D4 applies the same repeatability idea at evaluation time instead of duplicating the validation probe.
• RFC 012 (Harbor capture): relevant to open question 8, whether the first evaluation covers Harbor.
• RFC: GRPO recipes #924 (GRPO recipes, draft): this RFC keeps trainers out of core, consistent with its non-goals, and offers reasoning_gym_env as a partial answer to its open question 2 (cheap, but single-turn with no tool use).
• docs: add hf rl environment datasets rfc #795 (HF RL environment datasets): possible home for published openenv eval results (open question 4).
• RFC: Environment Services — brokered, budgeted access to external services (training, GPU jobs, evaluators, simulators) #1382 (environment services) and RFC 014 (RFC 014: Environment Services (design only) #1508): a per-episode spend reported by the environment is one source of the costs a trainer passes to update. posttrain_env (Add experimental posttrain_env prototype (RFC 014) #1500) ends an episode whose score job fails with a reward of None and still reports its spend; the samplers count that episode and its spend in episodes and cost, and leave its reward out of every estimate made from rewards.

Tier 1: Fixes Required

None: the PR changes only Markdown.

Tier 2: Alignment Discussion

Principle Conflicts

ALIGNMENT FLAG: Task metadata over HTTP

• Principle/RFC at stake: INVARIANTS.md, Architectural Invariant 4 (WebSocket for all environment communication, "Gym-like API + metadata"; HTTP deprecation in [PATCH] Deprecate HTTP in websockets #252, Deprecate HTTP /reset, /step, /state #1156)
• The concern: TaskAPIClient consumes the existing HTTP-only Task API routes, as documented in the Task API guide. The RFC adds no routes, but it adds a core consumer of them. D5 isolates this in one class so a later WebSocket move changes only TaskAPIClient.
• Suggested reviewer: @Darktex, @adithya-s-k

ALIGNMENT FLAG: Evaluation follows the collect pipeline

• Principle/RFC at stake: Principle 1 (minimize lifecycle deltas)
• The concern: Building the evaluation from the collect pipeline's pieces keeps training and evaluation on one path, but it inherits the pipeline's limits: openenv collect resolves only openspiel: and reasoning_gym: specs, CollectRunner passes no seed and records no failed episode (rollout step 5 adds both), and Harbor, which runs a whole agent per call, is not covered by the first version (D3, open question 8).
• Suggested reviewer: @Darktex, @burtenshaw

RFC Conflicts

ALIGNMENT FLAG: Task API helpers in core

• Principle/RFC at stake: Task API guide ("The core clients (EnvClient, SyncEnvClient) do not ship Task API methods, because task specs are environment-specific")
• The concern: The RFC adds a generic TaskAPIClient to openenv.core. D6 argues it keeps specs untyped and adds nothing to EnvClient, so the guide's reasoning still holds, but this reverses a documented choice and should be agreed explicitly.
• Suggested reviewer: @adithya-s-k

No conflicts found with the dual API boundary, rewards-in-environment, client-server separation, or "one env = one trajectory": samplers run on the orchestration side and only read rewards the environment computes, and a reward the environment could not give is left out of everything estimated from rewards and of the evaluation's success_rate, mean_reward and pass@k, never counted as zero, while its episode and its cost still count; a failure of the policy that reaches the environment gets the environment's own score.

Summary

• 0 mechanical issues to fix
• 2 alignment points for human review
• 1 RFC conflict to discuss

Disclosure

#1250 and #1252, cited in the RFC, are by one of the authors. This PR was prepared with AI assistance.

---

Note

Low Risk
Only Markdown is added; follow-on PRs would touch eval, collect pipeline, and core APIs, which the RFC flags for maintainer review (HTTP Task API, generic client in core).

Overview
This PR is documentation only: it adds RFC 013 (~540 lines) and a Reward & Evaluation index line in rfcs/README.md. No runtime or library code changes.

The RFC specifies trainer-side task sampling and in-training evaluation on top of the Task API: a TaskSampler protocol (SequentialSampler, UniformSampler, optional example CurriculumSampler), TaskAPIClient, EnvEvalHarness / compare_eval_results, openenv eval, and EvalSchedule. It defines how None rewards (environment unscored rollouts) affect sampler stats and eval metrics (error_rate, unscored_rate, infra_error_rate, checkpoint left_out tasks), drops cost from curriculum weights after multi-seed trials, and lays out a six-step rollout (guide + client, env Task API, samplers, harness/CLI/collect fixes, schedule + reference experiment). Preliminary GRPO/MLX results on reasoning_gym_env are included; implementation is deferred to follow-on PRs.

^{Reviewed by Cursor Bugbot for commit 693135e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 3 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit a95de40. Configure here.}

[tiagoperrupato]

We pushed a revision of the RFC with results on three more seeds; the list of changes is in the description.

The pilot's per-token result did not repeat. Difference to uniform sampling, in points of test success (500 questions), as the mean of the evaluations at 4 to 5 million generated tokens ("per token") and at steps 100 to 700 ("per episode"):

	Seed 0 (pilot)	Seed 1	Seed 2	Seed 3
Curriculum, per token	−7.7	+5.5	+18.3	+14.4
Curriculum, per episode	+2.3	+4.3	+9.7	+6.5
Token-weighted curriculum, first formula, per token		+1.2	−1.6	−1.7
Token-weighted curriculum, corrected, per token		+5.6	+15.2	+16.1
Token-weighted curriculum, corrected, per episode		+10.3	+10.1	+9.2

Per episode, seed 2 is read at steps 100 to 700 for each pair of runs that reached step 700. The rule we wrote before the run read every run of a seed only up to the step all of them reached; one run of seed 2 stopped at step 223, and that rule gives +3.9 for the curriculum, with the same conclusion. Three new seeds are a pattern, not a significance test.

What changed in the proposal:

• Weighting by token cost (cost="tokens") uses a corrected formula; the default, cost="episodes", weights by GRPO signal as before.
• Of the samplers, only the protocol and the two simple ones are proposed for core; the curriculum is offered as an example.
• The first version of the evaluation covers environments whose turns the trainer drives, and the gaps in the collect pipeline are listed in the rollout plan.

Three questions for the maintainers are in #1445. The trial's code and logs are at Matheus-F-Scatolin/OpenEnv@rfc-013-trial.

[rycerzes]

Thanks @Matheus-F-Scatolin @tiagoperrupato for the RFC and the trial data. The TaskSampler protocol, TaskAPIClient, the eval harness and its reproducibility checks look good to me. I'm requesting changes on the cost path only, with details inline:

1. Keep cost out of the sampling weights. I found no lab report or paper that divides a task's sampling weight by its cost. Labs that handle cost keep per-source shares and give slow sources more rollout capacity (MiMo-V2.6, LongCat-Flash-Thinking-2601, MiniMax M2), and budget-aware methods move rollouts toward hard tasks (Knapsack RL). With dollar costs from an environment (#1382), dividing would starve the expensive, hard tasks. Costs should be reported, and used for budgets and concurrency.
2. State the trainer hooks a sampler needs. It needs to draw prompts from sample(n) and to call update on every scored group before dynamic-sampling filters such as huggingface/trl#7609 drop the all-pass and all-fail groups. Aligning with prime-rl's TaskSampler would help, and CurriculumSampler can stay an example (open question 9).
3. Cite the trainer-side pieces rather than define them. Post-generation filtering and per-environment mixture shares are enforced by the trainer's rollout scheduler, and the group key already carries the metadata those shares need.

[tiagoperrupato]

Thanks @rycerzes for the careful review. 2a5bbf6 revises the RFC:

• Cost is out of the curriculum's weights (your point 1), and CurriculumSampler loses its cost argument. The samplers proposed here count the cost they are given and report signal per episode and per unit of cost.
• Trainer integration states the two hooks and the update-before-filter order, and what we found about TRL and asynchronous trainers (point 2). metrics() is added to the protocol; sample(n) and update stay, for the reasons in the thread on the trainer hooks.
• Shares across sources and the bias of asynchronous trainers are cited, not defined (point 3).
• Section 4 reports how far the signal formula over-predicts in our runs. Open question 5 takes your suggestion as a direction; open question 6 takes your answer.
• The reference experiment now compares uniform sampling with the curriculum only, on five seeds not used so far, with its test and reading windows written down before the runs.

On the sources: we read the five reports, huggingface/trl#7609 and prime-rl's sampler. For lab practice they support what you describe: none of them divides a task's sampling weight by its cost. Outside lab reports the idea does exist. Graves et al. (2017) divide a task's learning progress by the time a sample of it takes to process (the length of its longest input sequence), and two 2026 preprints that apply the idea to GRPO, DUET and Cost-Aware Learning, divide a usefulness term by the square root of the cost, to allocate rollouts within a batch and to sample update minibatches. Cost-aware Bayesian optimization also documents the failure you expect, in best-point search: a plain value/cost ratio keeps choosing cheap, low-value points (Lee et al., 2020; Astudillo et al., 2021). The RFC cites Graves et al., DUET and Cost-Aware Learning in Alternatives Considered; the last two allocate rollouts or minibatches, not tasks. One nuance on MiniMax-M2: it gives the instability of batches tilted toward short, easy tasks as the reason for its windowed FIFO, without a measurement, and that tilt comes from completion order in an asynchronous pipeline, not from a constant reweighting such as dividing by cost.

[rycerzes]

Thanks for addressing the issues @tiagoperrupato @Matheus-F-Scatolin
you can resolve the review conversation since they are outdated now

One suggestion: keep the RFC to OpenEnv's side. The prime-rl wrapper sketch (L155) and the TRL IterableDataset path (L257) describe those trainers' internals and would fit better in their repos or in the examples/ of rollout step 6. The two hooks (L255) and the notes on what stays with the trainer (L259, L261) are enough here.

[tiagoperrupato]

Thanks @rycerzes for the approval and the suggestion. 693135e takes it: the RFC no longer sketches a prime-rl wrapper (in TaskSampler) or the TRL IterableDataset path (in Trainer integration). The RFC keeps the note that the protocol has the shape of prime-rl's TaskSampler, and keeps the two hooks, the limits of asynchronous trainers and the shares across sources. Rollout step 6 still leaves a TRL example of the sampler for later, and what we found about that path stays in the thread on the trainer hooks for when the example is written.

693135e also states what RFC 014 (#1508) and posttrain_env (#1500) now depend on: what a reward of None means. It is a rollout the environment could not score because of a failure of its own, such as a failed score job. A failure of the policy that reaches the environment, such as a wrong or malformed answer in a valid tool call or hitting a limit the environment enforces (as posttrain_env's step and time limits do), is not one: the environment scores it, usually 0, and the trainer passes that score to update. It is the split you gave for posttrain_env on #1500: "None = our failure, left out by RFC 013; 0.0 = the policy hit a limit." For a None, samplers leave the reward out of everything they estimate from rewards and never count it as zero, but still count its episode and its cost, so a score job that fails after the agent's jobs were paid for still shows in cost and in signal groups per unit of cost. That matches RFC 014's "an episode that ends unscored still reports its spend". Evaluation leaves a None out of success_rate, mean_reward and pass@k and reports it separately as unscored_rate, while a failure of the policy counts as not solved. One passage may want the same wording: the posttrain_env README's "Trainers should drop such a rollout, not count it as zero". Under this revision a trainer passes such a rollout to update with its None reward and its spend, as the same README already asks for the spend ("retain that accounting even when reward is None"); rollout step 5 keeps such a rollout apart in the episode record, since today the harness helpers raise when the environment gave no reward.