feat(tier3): rebuild ATIF from OpenCode and Codex agent logs by mimran-khan · Pull Request #119 · NVIDIA/SkillEvaluator

mimran-khan · 2026-08-28T21:59:56Z

Fixes #118

When Harbor leaves trajectory.json empty, Tier 3 scoring used to fail with "No trajectory or reconstructible agent log" even though OpenCode and Codex tee structured JSON to opencode.txt and codex.txt.

This adds parsers for OpenCode run --format=json events (text + completed tool_use records) and wires them into load_trajectory_with_fallback after the existing claude/cursor/cline fallbacks. Codex reuses the same parser when every tee line is JSON. Non-dict tool inputs are ignored so raw strings are not treated as shell commands.

Harbor task bundles already copy log_converters.py through adapter.py, so generated verifiers pick this up automatically.

Distinct from #110, which is about exec steps inside an existing trajectory.

Test plan

pytest tests/tier3/test_log_converters.py
pytest tests/test_harbor_collector_runtime_failures.py

Parse OpenCode JSON stream events into synthetic trajectories when trajectory.json is empty, and reuse the parser for structured Codex tee logs. Fail closed on non-dict tool inputs. Fixes NVIDIA#118 Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>

chrisknvidia · 2026-08-29T00:18:38Z

+    if not lines:
+        return None
+    if all(line.startswith("{") and line.endswith("}") for line in lines):
+        synth = synthetic_trajectory_from_opencode_json(text)


[P1] codex exec --json does not emit OpenCode tool_use events. A successful Codex CLI 0.142.5 run emitted thread.started, turn.started, item.completed with item.type="agent_message", and turn.completed; passing those JSONL records here returns None. Harbor also tees 2>&1, so one warning line prevents this branch entirely. Please parse Codex's own item events, tolerate non-JSON stderr, and add a fixture captured from real Codex stdout instead of reusing the OpenCode schema.

Fixed: added a Codex exec --json parser for item.completed / agent_message events, stderr-tolerant JSONL scanning, and fixture tests. OpenCode-shaped JSONL still works as a fallback.

This is still not fixed at the current head. Codex exec JSON emits top-level item.completed events; agent messages use item.text, and shell calls are separate command_execution items with command and aggregated_output fields. Feeding a successful stream in that shape still returns no synthetic trajectory, so an empty trajectory file collapses to the no-reconstructible-log result. Please parse the published ThreadEvent schema and cover it with a captured CLI fixture.

This should be fixed now. synthetic_trajectory_from_codex_json handles top-level item.completed events, reads agent_message from item.text, and maps command_execution items with command plus aggregated_output into bash tool calls. Added fixture coverage for that Codex thread shape.

The Codex item.completed and command_execution parsing from the earlier round is still in place at this head. I also merged main (#111) so we pick up the upstream codex_tool_call_normalizer without dropping the log-converter fallback path.

Codex tee logs use type=item / agent_message events, not OpenCode tool_use. Add a dedicated parser with stderr-tolerant JSONL scanning, keep OpenCode-shaped JSONL as a fallback, and normalize shell calls to bash for skill checks. Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>

mimran-khan · 2026-08-30T07:16:44Z

Codex parser fixes are pushed from the earlier round. Ready for re-review when you have time.

rng1995 · 2026-08-31T16:25:05Z

            meta["note"] = "Synthetic ATIF from Cline CLI JSONL log"
            return synth, meta

+    opencode_path = logs / "opencode.txt"


[P2] This fallback is not reachable from eval-dataset refinement with collected OpenCode results. Trajectory discovery never includes opencode in agent_priority, so a persisted opencode/with-skill/trials//opencode.txt run is skipped wholesale. I reproduced an empty discovery result for that layout while the same fixture under a listed agent is found. Please add OpenCode to trajectory discovery and cover the public refinement path with an integration test.

opencode is in agent_priority now, and I added a regression that discovers trajectories from opencode.txt when trajectory.json is missing.

OpenCode is in the agent_priority list and load_trajectory_with_fallback already probes opencode.txt. The Harbor trial id normalization fix should also let refine match trajectories discovered from OpenCode runs.

rng1995

The current head still does not reconstruct real Codex exec events, and the new OpenCode fallback is not connected to the collected-results discovery path. Focused converter and refinement tests passed (95 tests), with Ruff and diff checks clean. Requesting changes for the two reproducible integration gaps noted in the review threads.

…ries Merge main and handle Codex exec JSON item.completed events with agent_message text and command_execution items. Include opencode in trajectory discovery so refinement can rebuild ATIF from opencode.txt when trajectory.json is missing.

…an-khan/SkillEvaluator into feat/opencode-atif-fallback

chrisknvidia · 2026-09-03T10:21:08Z

+    agent_priority = [
+        "claude-code",
+        "cursor-cli",
+        "opencode",


[P1] Normalize Harbor trial names before matching case IDs. Harbor names trial directories as <case-id>__<random-suffix> (and can truncate long IDs), but this path stores the entire directory name as the trajectory key while refinement later does exact trajectories.get(case["id"]) lookups. Replaying this head against a retained 26-case OpenCode Harbor run discovered 19 valid trajectories but matched 0/26 cases and changed none; config.json.task.path mapped all 19 correctly. Please derive the canonical case ID from Harbor metadata with a validated fallback, and cover a suffixed/truncated real-layout fixture through the refinement path.

Fixed at 644a06a (head 05f7b62 after merging main). _case_id_from_trial_dir now resolves from reward entry_id, then result.json task metadata, before falling back to the full folder name so Harbor trial dirs map to eval case ids during refine.

chrisknvidia · 2026-09-03T10:21:21Z

+            steps.append(step)
+            continue
+
+        if item_type == "command_execution":


[P1] Preserve the other completed Codex action items. The Codex 0.153.0 ThreadItem schema also emits file_change and mcp_tool_call, but this parser stops at agent messages and command_execution. A real 0.153.0 file-write run contained a completed file_change while this function produced no write call; a file/MCP write can therefore disappear from downstream file-change evidence, behavior grading, and trace-level security checks. Please map those action items into ATIF calls/observations, preserve status/error and command exit_code, and add captured current-CLI fixtures.

Added parsing for Codex file_change and mcp_tool_call completed items in synthetic_trajectory_from_codex_json, with tests in test_log_converters.py. This sits alongside the existing command_execution and agent_message handling.

chrisknvidia · 2026-09-03T10:21:30Z

+
+
+def _opencode_output_text(state: dict[str, Any]) -> str:
+    output = state.get("output")


[P2] Keep failed OpenCode tool outcomes in the observation. OpenCode error events put the failure text in state.error rather than state.output, so this helper returns an empty string and the synthetic ATIF records the attempted call with no indication it failed. In a retained real OpenCode log, all 6 status=error calls became empty observations, leaving downstream recovery and accuracy evidence unable to distinguish failure from a successful call with no output. Please fall back to state.error when output is absent, retain the terminal status, and add an error-state fixture.

_opencode_output_text now falls back to state.error when output is empty, so failed tool calls keep their failure text in the observation. Covered by test_opencode_tool_error_uses_state_error_observation.

@chrisknvidia @rng1995 All three Sep 3 items are addressed at 05f7b62 (main merged, including #111). Fork CI needs approval before checks run. Ready for re-review when you have time.

Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>

Resolve trial case ids from reward/result metadata before refine lookup. Parse Codex file_change and mcp_tool_call events, and keep OpenCode tool failures in observations via state.error. Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>

Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>

…llback

mimran-khan requested review from chrisknvidia, mosheabr, rmalani-nv and rng1995 as code owners August 28, 2026 21:59

chrisknvidia reviewed Aug 29, 2026

View reviewed changes

Merge branch 'main' into feat/opencode-atif-fallback

1c13159

rng1995 reviewed Aug 31, 2026

View reviewed changes

rng1995 requested changes Aug 31, 2026

View reviewed changes

mimran-khan added 2 commits August 31, 2026 22:31

Merge branch 'feat/opencode-atif-fallback' of https://github.com/mimr…

935c154

…an-khan/SkillEvaluator into feat/opencode-atif-fallback

chrisknvidia reviewed Sep 3, 2026

View reviewed changes

mimran-khan and others added 7 commits September 3, 2026 15:56

merge main into feat/opencode-atif-fallback

befde1c

Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>

merge main into feat/opencode-atif-fallback

05f7b62

Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>

Merge branch 'main' into feat/opencode-atif-fallback

28e9e0f

Merge remote-tracking branch 'origin/main' into feat/opencode-atif-fa…

e5bcc4e

…llback

chore: reconcile fork feat/opencode-atif-fallback

b7bbda8

chore: merge main into feat/opencode-atif-fallback

97ec837



		def _opencode_output_text(state: dict[str, Any]) -> str:
		output = state.get("output")

Uh oh!

Conversation

mimran-khan commented Aug 28, 2026

Test plan

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mimran-khan commented Aug 30, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rng1995 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants