feat(evaluator): taskset references for agent evaluation + Tasks/Tasksets docs#779
Open
SandyChapman wants to merge 3 commits into
Open
feat(evaluator): taskset references for agent evaluation + Tasks/Tasksets docs#779SandyChapman wants to merge 3 commits into
SandyChapman wants to merge 3 commits into
Conversation
Contributor
|
Add a Manage Tasks & Tasksets page covering the stored task/taskset entities added in the AALGO-307 stack: concepts, create/retrieve/list/ delete via the nemo_platform SDK (client.evaluator.tasks / .tasksets), field references, workspace/project handling, and the underlying REST API. Registers the page in the published Fern nav after SDK Resources. Fills a documented gap: there was no user-facing doc for task/taskset management. Signed-off-by: Sandy Chapman <schapman@nvidia.com>
SandyChapman
force-pushed
the
schapman/evaluator-tasks-taskset-docs-7e3261
branch
from
July 21, 2026 11:35
bc9f1ae to
fb31f70
Compare
SandyChapman
changed the title
…path AgentEvalInputSpec.tasks becomes a union: an inline list of tasks or a TasksetRef naming a stored taskset. During to_spec a taskset reference is loaded and its member tasks are expanded into inline task DTOs, then the existing metric-ref resolution runs — so a taskset-driven run hydrates to the same canonical AgentEvalSpec as an inline one. - Add TasksetRef (schemas.py) and task_refs.py (resolution helper mirroring metric_refs.py): loads the taskset + member tasks, maps each stored task to an AgentEvalTaskInput, and raises clear errors for a missing taskset, a missing/deleted member, an empty taskset, duplicate expanded ids, or a local run with no entity store. - Stored tasks carry no grader-only reference field, so taskset-driven tasks run with an empty reference (documented). - Document the run-over-a-taskset flow in the Manage Tasks & Tasksets page. - Regenerate the plugin OpenAPI spec for the new union + TasksetRef schema. Tests: unit coverage for resolution + the union field; verified live end to end that a TasksetRef resolves through to_spec into a hydrated AgentEvalSpec with inline metrics. Signed-off-by: Sandy Chapman <schapman@nvidia.com>
SandyChapman
force-pushed
the
schapman/evaluator-tasks-taskset-docs-7e3261
branch
from
July 21, 2026 12:39
09c8023 to
666f57e
Compare
Integration test (opt-in, RUN_AGENT_EVAL_INTEGRATION=1): stores a numeric metric + two tasks + a taskset via the SDK, then submits an agent-evaluate job whose `tasks` is a TasksetRef (no inline tasks) against a real spun platform with an IGW mock model. Proves the server resolves the taskset -> both member tasks -> each task's stored MetricRef -> inline, runs both, and scores them: asserts nan_count == 0, count == 2, mean == 1.0 (both members ran and scored, no failed samples). Adds a numeric _OutputScoreMetric (continuous_score) so the run aggregate carries a real count/mean; a boolean output would land in nan_count and obscure whether scoring actually succeeded. Signed-off-by: Sandy Chapman <schapman@nvidia.com>
SandyChapman
marked this pull request as ready for review
July 21, 2026 13:29
Contributor
📝 WalkthroughWalkthroughChangesTaskset references are now supported in agent evaluation inputs. Stored tasksets resolve into inline tasks and stored metric references before evaluation, with validation for missing, empty, or duplicate task entries. Documentation, OpenAPI schemas, navigation, unit tests, and integration coverage were added. Taskset-backed agent evaluations
Sequence Diagram(s)sequenceDiagram
participant Client
participant AgentEvalJob
participant EntityClient
participant MetricResolution
Client->>AgentEvalJob: submit AgentEvalInputSpec with TasksetRef
AgentEvalJob->>EntityClient: load taskset and member tasks
EntityClient-->>AgentEvalJob: stored task entities
AgentEvalJob->>MetricResolution: resolve metrics for expanded tasks
MetricResolution-->>AgentEvalJob: canonical evaluation spec
AgentEvalJob-->>Client: evaluation result
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
plugins/nemo-evaluator/src/nemo_evaluator/task_refs.py (1)
43-93: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winParallelize member-task fetches.
resolve_taskset_refawaitsentity_client.get(TaskEntity, ...)once per member sequentially. For a taskset with many members this serializes N entity-store round trips. Dedup detection doesn't depend on fetch order, so gather the fetches concurrently.⚡ Proposed fix: fetch members concurrently
+import asyncio + async def resolve_taskset_ref( ref: TasksetRef, *, workspace: str, entity_client: EntityClient | None, ) -> list[AgentEvalTaskInput]: ... - tasks: list[AgentEvalTaskInput] = [] - seen_ids: set[str] = set() - for task_ref in taskset.tasks: - task_workspace, task_name = parse_entity_ref(task_ref.root, ref_workspace) - try: - entity = await entity_client.get(TaskEntity, name=task_name, workspace=task_workspace) - except EntityNotFoundError as exc: - raise ValueError( - f"Task '{task_ref.root}' referenced by taskset '{ref.root}' was not found; " - "the stored task may have been deleted after the taskset was created." - ) from exc - if entity.name in seen_ids: - raise ValueError( - f"Taskset '{ref.root}' expands to more than one task named '{entity.name}'; " - "task ids must be unique within an evaluation." - ) - seen_ids.add(entity.name) - tasks.append(_entity_to_task_input(entity)) - return tasks + async def _fetch(task_ref: TaskRef) -> TaskEntity: + task_workspace, task_name = parse_entity_ref(task_ref.root, ref_workspace) + try: + return await entity_client.get(TaskEntity, name=task_name, workspace=task_workspace) + except EntityNotFoundError as exc: + raise ValueError( + f"Task '{task_ref.root}' referenced by taskset '{ref.root}' was not found; " + "the stored task may have been deleted after the taskset was created." + ) from exc + + entities = await asyncio.gather(*(_fetch(task_ref) for task_ref in taskset.tasks)) + tasks: list[AgentEvalTaskInput] = [] + seen_ids: set[str] = set() + for entity in entities: + if entity.name in seen_ids: + raise ValueError( + f"Taskset '{ref.root}' expands to more than one task named '{entity.name}'; " + "task ids must be unique within an evaluation." + ) + seen_ids.add(entity.name) + tasks.append(_entity_to_task_input(entity)) + return tasks🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-evaluator/src/nemo_evaluator/task_refs.py` around lines 43 - 93, Update resolve_taskset_ref to fetch all member TaskEntity records concurrently rather than awaiting entity_client.get inside the sequential loop. Build the member-reference requests first, use concurrent gathering while preserving each taskset.tasks order, then retain the existing not-found error context and seen_ids validation when converting the fetched entities.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@plugins/nemo-evaluator/src/nemo_evaluator/task_refs.py`:
- Around line 43-93: Update resolve_taskset_ref to fetch all member TaskEntity
records concurrently rather than awaiting entity_client.get inside the
sequential loop. Build the member-reference requests first, use concurrent
gathering while preserving each taskset.tasks order, then retain the existing
not-found error context and seen_ids validation when converting the fetched
entities.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: bc7a2b1c-9cd7-49a9-aca1-2fa3172c71b7
📒 Files selected for processing (12)
docs/evaluator/manage-tasks-tasksets.mdxdocs/fern/versions/latest.ymlplugins/nemo-evaluator/openapi/openapi.yamlplugins/nemo-evaluator/src/nemo_evaluator/api/schemas.pyplugins/nemo-evaluator/src/nemo_evaluator/jobs/agent_evaluate.pyplugins/nemo-evaluator/src/nemo_evaluator/jobs/agent_spec.pyplugins/nemo-evaluator/src/nemo_evaluator/jobs/evaluate.pyplugins/nemo-evaluator/src/nemo_evaluator/jobs/metric_resolution.pyplugins/nemo-evaluator/src/nemo_evaluator/task_refs.pyplugins/nemo-evaluator/tests/integration/test_agent_evaluate_job.pyplugins/nemo-evaluator/tests/test_agent_evaluate.pyplugins/nemo-evaluator/tests/test_task_refs.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two related changes for stored evaluator Tasks & Tasksets:
1. Feature — taskset references on the agent-eval submit path
Previously an agent evaluation could only be submitted with an inline list of tasks; stored tasksets were CRUD-only with no consumer. This makes
AgentEvalInputSpec.tasksa union:During
AgentEvalJob.to_spec, aTasksetRefis loaded and its member tasks are expanded into inline task DTOs, then the existing metric-ref resolution runs — so a taskset-driven run hydrates to the same canonicalAgentEvalSpecas an inline one.TasksetRef(api/schemas.py) andtask_refs.pyresolution helper (mirrorsmetric_refs.py).referencefield (it lives only on inlineAgentEvalTaskInput), so taskset-driven tasks run with an emptyreference. Best for metrics that score output directly.tasksbecomesanyOf [TasksetRef, array]).2. Docs — Manage Tasks & Tasksets
New page covering CRUD for stored tasks/tasksets via
client.evaluator.tasks/.tasksets, field references, the REST API, and a Run an evaluation over a taskset section demonstrating the feature above. Registered in the Fern nav after SDK Resources.Testing
plugins/nemo-evaluator/tests/test_task_refs.py(new) — resolution unit coverage;test_agent_evaluate.py— union-field schema tests.ruff check/ruff format --check/ty check(CI script) clean on changed files.make docs-check+make docs-broken-linkspass.AgentEvalInputSpec(tasks=TasksetRef(...)), and confirmedto_specresolves to a hydratedAgentEvalSpec(2 tasks, inline metrics); unknown taskset errors cleanly. Also confirmed the doc's create/list/delete examples and the inline-metric normalization behavior end to end.🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Bug Fixes