Decide how to store and share model reasoning traces

[gonzalobenegas]

Context

Evaluation runs preserve the visible answer in runs/*/results.jsonl and the full provider payload/reasoning trace in the content-addressed data/response_cache/. Those paths are intentionally Git-ignored. During the Nemotron 3 Ultra evaluation tracked in #6, we decided the traces are scientifically valuable because reproducing them can be expensive in both wall time and API cost.

This issue is for the storage decision only. It should remain separate from model evaluation and response-viewer implementation.

Current evidence

An early sample of three Nemotron cache entries measured:

• 78,875 bytes uncompressed
• 11,474 bytes as a gzip-compressed tar archive
• 14.5% compressed/raw ratio
• straight-line 54-response projection: ~1.35 MiB raw / ~0.20 MiB compressed

Later responses include reasoning traces around 25k characters and individual calls taking roughly nine minutes, so storage size is currently modest while regeneration cost is not.

Storage properties we need

• Preserve the exact successful provider payload and reasoning trace.
• Bind traces to the run config, dataset fingerprint, cache key, and visible result row.
• Verify integrity with SHA-256 checksums.
• Avoid committing the mutable shared cache, locks, partial state, unrelated models, or regenerated gold answers.
• Let collaborators inspect traces from another clone/session.
• Ideally allow exact cache restoration so a preserved response prevents another provider call.
• Keep API credentials and secrets out of every artifact.

Options to compare

1. Compressed immutable artifacts in Git

   • One .tar.gz/.tar.zst per completed run plus a readable manifest.
   • Lowest operational overhead and likely appropriate during exploration.
   • Binary artifacts do not diff and will eventually bloat repository history.

2. Git LFS

   • Keeps large blobs outside ordinary Git objects while retaining repository references.
   • Adds quota, tooling, and clone/setup considerations.

3. GitHub Release assets

   • Durable downloadable bundles without bloating Git history.
   • Awkward lifecycle/tag semantics for frequent exploratory runs.

4. Content-addressed object storage

   • Store each compressed cache entry once under its cache key; run manifests reference shared blobs.
   • Best deduplication and long-term scale, but introduces credentials, lifecycle policy, availability, and implementation work.

Proposed exploratory default

Until measurements show otherwise, check in one immutable compressed bundle per completed run under experiments/model-traces/, accompanied by a readable manifest and checksums. Do not check in data/response_cache/ itself. Revisit LFS/object storage when a single bundle or the accumulated trace history becomes inconvenient enough to affect normal clones and reviews.

Questions / decision criteria

• Expected number of models, reasoning-effort variants, repeats, and dataset versions.
• Actual compressed size distribution after complete runs, especially 64k-token traces.
• Whether traces need private access or have provider redistribution constraints.
• Desired one-command export, verify, inspect, and restore workflow.
• Whether the response visualizer should read archives directly or only extracted/local cache entries.
• Thresholds that trigger migration away from ordinary Git.

Completion criteria

• Measure at least one complete reasoning-model run.
• Record the chosen exploratory storage policy in repository docs.
• Define a versioned manifest schema and deterministic export layout.
• Provide integrity verification and cache-restore instructions.
• Confirm the response visualizer can inspect preserved traces.
• Document migration triggers and the next preferred backend if Git stops being suitable.

[gonzalobenegas]

Updated measurement from the active Nemotron run (10 successful cache entries):

• exact cache payloads: 853,764 bytes raw
• deterministic tar+gzip sample: 195,737 bytes
• reasoning text: 266,204 characters
• append-only results ledger (including failures/retries): 22,997 bytes

A simple linear projection to all 54 smoke renders is about 4.5 MiB raw / 1.0 MiB compressed, though the distribution is uneven (one mechanistic trace was much larger/slower than the early geometry samples).

This still supports the exploratory default proposed above: commit one immutable compressed per-run archive plus a readable manifest/checksum, not the mutable shared cache. Revisit LFS/releases/object storage only when actual archives cross the agreed threshold or accumulate enough history to burden clones.

The review UI on issue #6 now reads either the live cache or an unpacked archive cache via --response-cache, so this storage choice does not require a second visualization format.

[gonzalobenegas]

Complete-run measurement and immediate preservation status

We now have concrete measurements from the completed OpenRouter exploration:

• The entire live shared cache is 8.4 MiB across 309 files.
• The selected four-model portable bundle is 2.5 MiB compressed.
• It contains the exact smoke dataset needed by the reviewer, four run ledgers and configs, and 150 cache entries referenced by successful results.
• Luna: 54/54 successful rows and 54/54 referenced cache entries preserved.
• DeepSeek V4 Flash high: 54/54 successful rows and 54/54 reasoning traces preserved.
• Nemotron Ultra: 32/32 successful rows preserved with cache entries.
• Nemotron Lightning: 14 successful rows, but only 10 corresponding cache entries were still present. The four visible answers remain in the ledger; their full traces were already absent before export.
• Bundle SHA-256: 338cd64f498b086397132769ef2e87f5026d2ffbc729bfeb3eea30d1eb5cb3e7.

The bundle includes MANIFEST.sha256, a versioned bundle.json, and instructions. It was extracted into a clean directory, every checksum verified, and the existing multi-run viewer successfully loaded all four models plus a stored DeepSeek reasoning trace.

The exact review command is already supported after extraction:

structural-reasoning review \
  --dataset <bundle>/data/datasets/smoke \
  --responses <bundle>/runs/deepseek-v4-flash-high \
  --responses <bundle>/runs/gpt-5.6-luna-medium \
  --responses <bundle>/runs/nemotron-3.5-lightning-free \
  --responses <bundle>/runs/nemotron-3-ultra-free \
  --response-cache <bundle>/data/response_cache

The current archive is deliberately marked private/local because it contains rendered prompts and gold.jsonl. It should be copied to an approved off-node internal location before this EC2 experiment is torn down. Whether released archives should instead be committed, uploaded as Release assets, or published through a static site remains the decision tracked here.

The separate question of whether to publish a static GitHub Pages explorer is tracked independently so it does not block immediate preservation and CLI-based review.

[gonzalobenegas]

Off-node snapshot available

The verified 2.6 MiB portable bundle is now attached to the draft release
trace-bundle-2026-08-14-v2.

It is deliberately a draft, not a public release, because it contains exact
rendered prompts and gold.jsonl. Repository collaborators can download it with:

gh release download trace-bundle-2026-08-14-v2 \
  --repo Open-Athena/pdbthink \
  --pattern 'pdbthink-openrouter-review-2026-08-14-v2.tar.gz'

The release notes and bundled README contain checksum verification and the full
four-model structural-reasoning review command. The archive also includes the
exact viewer source and installation steps, so Tim does not need the unmerged
working branch. This gives the current paid traces an off-node copy without
deciding the final publication/storage policy.

Static/public viewer discussion remains separate in #8.