fix(evaluator-sdk): persist metric diagnostics on the success path

[SandyChapman]

What

The evaluator's success path (_score_metric) built the completed AgentEvalTaskScore with outputs=... but omitted diagnostics, so a metric's own diagnostics were silently dropped on success — while the failure path (_failed_metric_score) recorded them. This converts MetricResult.diagnostics → AgentEvalDiagnostic(severity=info, source=metric_type) on the success path too, so they persist to scores.jsonl.

Why

Metrics that surface component-level detail (e.g. per-criterion LLM-judge verdicts + reasoning) had that signal computed and then discarded on success. This is the mechanism for surfacing it.

Tests

New test_success_metric_diagnostics_are_persisted; the existing diagnostics == [] assertion for a no-diagnostics metric still holds. Includes the re-vendored SDK mirror (sdk/python/nemo-platform/...).

Summary by CodeRabbit

• Bug Fixes
  • Successful metric evaluations now retain diagnostic information in the resulting score, including severity, message, source, and details.
  • Diagnostic details are preserved consistently, with empty details provided when none are available.

[coderabbitai]

📝 Walkthrough

Walkthrough

Successful metric diagnostics are now converted into AgentEvalDiagnostic entries on AgentEvalTaskScore, with a regression test covering severity, message, source, and details.

Changes

Metric diagnostics persistence

Layer / File(s)	Summary
Map diagnostics into completed scores packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/evaluator.py	_score_metric persists metric diagnostics with INFO severity, preserving message and source and defaulting missing details to {}.
Validate diagnostic persistence packages/nemo_evaluator_sdk/tests/agent_eval/test_evaluator.py	Adds a diagnostic-emitting metric and verifies that its diagnostic fields are present in the aggregated score.

Possibly related PRs

• NVIDIA-NeMo/nemo-platform#687: Adds and propagates MetricDiagnostic through core scoring and execution plumbing.

Suggested reviewers: arpitsardhana, ngoncharenko

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: persisting metric diagnostics on the success path.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch persist-metric-diagnostics/schapman

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

packages/nemo_evaluator_sdk/tests/agent_eval/test_evaluator.py (1)

454-468: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick win

Exercise the scores.jsonl persistence path.

This test only inspects result.scores; it never configures output_dir or reads scores.jsonl. Add a temporary output directory and assert the serialized diagnostic fields so a JSONL persistence regression cannot pass unnoticed.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/nemo_evaluator_sdk/tests/agent_eval/test_evaluator.py` around lines
454 - 468, Add an output directory to
test_success_metric_diagnostics_are_persisted using the test’s
temporary-directory fixture, then read the generated scores.jsonl and assert the
persisted diagnostic’s severity, message, source, and details fields. Keep the
existing in-memory score assertions while ensuring the test exercises the JSONL
serialization path.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@packages/nemo_evaluator_sdk/tests/agent_eval/test_evaluator.py`:
- Around line 454-468: Add an output directory to
test_success_metric_diagnostics_are_persisted using the test’s
temporary-directory fixture, then read the generated scores.jsonl and assert the
persisted diagnostic’s severity, message, source, and details fields. Keep the
existing in-memory score assertions while ensuring the test exercises the JSONL
serialization path.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: c71f9196-94f8-48c9-8b8e-9a4c6c4b03e6

📥 Commits

Reviewing files that changed from the base of the PR and between 5a23930 and 9efa89d.

⛔ Files ignored due to path filters (1)

• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/evaluator.py is excluded by !sdk/**

📒 Files selected for processing (2)

• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/evaluator.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_evaluator.py

[github-actions]

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	26385/34033	77.5%	61.8%
Integration Tests	15172/32658	46.5%	18.7%