feat(artifact): model-level multi-run artifacts, append and merge#10
Merged
Conversation
Namespace dimension primary keys per run (run_id/name) so multiple runs coexist in one model-level SQLite file. Writers append to an existing artifact instead of deleting it, and re-writing the same run directory replaces that run. bench run gains --artifact for pointing batches at runs/models/<model>.sqlite. artifact check accepts one or more runs, each with exactly one original and one normalized spec. New localperf artifact merge --into combines artifacts: text ids rescope (prefix-if-missing handles old single-run sources), integer ids shift by the destination's max per referenced table, duplicate runs are skipped, and the destination is checked after every merge. The report loads every run (latest as header, all in a Runs section), aggregates repeated points across runs like repeats, and keeps run provenance on per-repeat rows. Implements phase 5 of the reporting completeness plan.
profiles.engine_id now stores the namespaced engine id; keying the served-model map by bare engine name silently disabled the declared-vs-served mismatch check for newly written artifacts. Key by engine id, which matches in both old and namespaced artifacts, and cover the namespaced path in the multi-run report test.
Appending refuses to replace a run recorded from a different run directory: run ids are directory basenames, and a basename collision is not a retry. The run row records its absolute run directory in labels_json for the comparison. Merge validates every source before touching the destination and removes a freshly created destination on failure, so a bad source cannot strand an empty artifact.
…ites Merge classifies same-id runs by provenance (created_at plus recorded run directory): matching provenance is an idempotent skip, differing provenance is a refused collision instead of a silent drop. The writer removes a freshly created artifact when its first write fails so retries do not hit a stranded schema-only file.
Runs written before the provenance label carry no run_dir; re-running the same run id over them keeps the old replace behavior instead of refusing as a collision. Remove the now-unused insertMetadata helper that failed lint.
No legacy allowance: a same-id run without a recorded run directory has unknown provenance and is refused, not silently replaced. Pre-cutover artifacts are out of scope per the plan; re-run them under the new format.
Member
Author
|
Final report: phase 5 implemented with strict cutover. Codex review ran 5 rounds; 7 findings fixed (engine-id keying for model mismatch, merge provenance collisions, failed-write and failed-merge cleanup, basename collision guard). The last finding (allow replacing legacy runs without a run_dir label) is deliberately rejected: cutover policy, no backward compatibility — unknown-provenance runs are refused and pre-cutover artifacts get re-run. Gates green locally (test/vet/gofmt/dry/crap/golangci-lint/simpledoc) and CI green. Merging. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The docs promised model-level artifacts (all runs for one model in
runs/models/<model>.sqlitewith one HTML report), but the writer deleted any existing file, the checker required exactly one run, and the report read a single run.This change wires that promise end to end: runs append into a shared artifact, a new merge command combines existing ones, and the report renders every run with cross-run aggregation.
It implements phase 5 of
docs/2026-07-02-reporting-completeness-plan.md.What Changed
The blocker was that dimension primary keys were bare names (profile "8k" from two runs collides), so appending was structurally impossible.
<run_id>/<name>) in the artifact writer;namecolumns stay bare for display. Not a schema change.artifact.CreateOrAppendopens an existing artifact for another run instead of deleting it; re-writing the same run directory replaces that run via cascade delete.bench run --artifactpoints batches at the model-level path.artifact checkaccepts one or more runs, each required to carry exactly one original and one normalized spec.localperf artifact merge --into <dst> <src>...: text ids rescope with a prefix-if-missing rule (old single-run and new namespaced sources both merge), integer ids shift by the destination's max per referenced table, duplicate runs are skipped so merges are idempotent, and the destination is checked after every merge.--artifactandmergeworkflows, the artifact format doc documents multi-run validation and id conventions, and the plan doc gains the phase 5 section.Testing
I ran the full gates plus a CLI-level acceptance of the exact documented workflow.
go test ./...,go vet, gofmt, simpledoc, slophammer check/dry/crap — all green.bench run --dry-run --artifactbatches appended plus oneartifact mergeof a separate single-run artifact produced a 3-run model artifact that passesartifact checkand renders one HTML report listing all three batches.Risks
Append semantics change the writer's contract: the artifact at
--artifactis no longer replaced wholesale.Re-running the same run directory intentionally replaces that run's rows, which is the retry behavior the sweep doc asks for.
🤖 Generated with Claude Code