Create refs for checkpoints#1397
Conversation
Implements the core of the hidden `entire checkpoint migrate-refs` command in a new file: tree ref naming, v1 branch checkpoint listing, resumable TSV cache, existing-ref snapshot, parallel skip-if-correct processing with progress, batched `git update-ref --stdin` writes, and the two-phase orchestrator with resume + dry-run. Covers Tasks 1-7 of the implementation plan. Cobra wiring (Task 8) and final verification (Task 9) follow separately. Assisted-by: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Paulo Gomes <paulo@entire.io> Entire-Checkpoint: 06f8b5a9bc39
Make the feeder goroutine select on ctx.Done() to close the goroutine leak window, and have processEntries return ctx.Err() so a cancelled run exits non-zero instead of printing a success summary. Add tests for cancellation and multi-batch ref writes. Assisted-by: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Paulo Gomes <paulo@entire.io> Entire-Checkpoint: d7621d86cfe0
Add newCheckpointMigrateRefsCmd (hidden) with --workers/--cache-file/ --refresh/--dry-run flags and register it under the checkpoint group. RunE resolves the repo root and defaults the cache file under the git common dir. Assisted-by: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Paulo Gomes <paulo@entire.io> Entire-Checkpoint: 08873c5d0aec
Compares resolving a single checkpoint's metadata subtree the previous way (v1 branch -> commit -> root tree -> shard navigation) against the new per-checkpoint tree ref. Warm (shared handle) and cold (fresh repo open per op) variants. Skips when the target repo is absent so it never runs in CI; override path via ENTIRE_BENCH_CHECKPOINTS_REPO. Assisted-by: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Paulo Gomes <paulo@entire.io> Entire-Checkpoint: bffa18d296ca
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 06be3c7. Configure here.
| return nil, err | ||
| } | ||
| fmt.Fprintf(opts.out, "Reusing cached list: %d checkpoints (%s)\n", len(entries), opts.cacheFile) | ||
| return entries, nil |
There was a problem hiding this comment.
Empty cache blocks migration
High Severity
After a walk finds zero checkpoints, loadOrBuildList still writes the cache file. Later runs treat any existing cache as authoritative and skip walking entire/checkpoints/v1, so newly added checkpoints never get tree refs unless the user deletes the cache or passes --refresh.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 06be3c7. Configure here.
| out: cmd.OutOrStdout(), | ||
| progress: cmd.ErrOrStderr(), | ||
| }) | ||
| return err |
There was a problem hiding this comment.
Ctrl+C prints cancellation error
Low Severity
The migrate-refs command returns context cancellation errors directly to Cobra, so interrupting a long run can print a noisy “context canceled” (or wrapped “process checkpoints”) message instead of exiting quietly like other CLI commands.
Additional Locations (1)
Triggered by learned rule: Map context.Canceled to NewSilentError on user cancellation
Reviewed by Cursor Bugbot for commit 06be3c7. Configure here.
There was a problem hiding this comment.
Pull request overview
This PR introduces a hidden maintenance command to backfill per-checkpoint git refs (pointing directly at each checkpoint’s metadata subtree) to avoid repeatedly walking the entire/checkpoints/v1 branch tree, and adds unit tests + opt-in benchmarks to validate and measure the approach.
Changes:
- Adds hidden
entire checkpoint migrate-refscommand to enumerate checkpoints onentire/checkpoints/v1, compute missing/stalerefs/entire/checkpoints/<ab>/<rest>/treerefs, and update them in batched atomic transactions viagit update-ref --stdin. - Adds comprehensive unit tests covering list-building, caching behavior, idempotency, dry-run mode, and ref writing.
- Adds local-only benchmarks comparing the old “walk v1 tree” resolver vs the new “tree ref” resolver.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| cmd/entire/cli/checkpoint_migrate_refs.go | Implements the migrate-refs command logic, including checkpoint enumeration, caching, concurrency, and batched ref updates. |
| cmd/entire/cli/checkpoint_migrate_refs_test.go | Adds end-to-end and unit tests for list building, caching, ref updates, and command wiring. |
| cmd/entire/cli/checkpoint_migrate_refs_bench_test.go | Adds opt-in benchmarks comparing v1-tree walking vs per-checkpoint tree refs for subtree resolution. |
| cmd/entire/cli/checkpoint_group.go | Registers the new hidden migrate-refs subcommand under entire checkpoint. |
| in := make(chan checkpointEntry) | ||
| out := make(chan refUpdate) | ||
| var processed int64 | ||
|
|
||
| var wg sync.WaitGroup | ||
| for range workers { | ||
| wg.Go(func() { | ||
| for e := range in { | ||
| if ctx.Err() != nil { | ||
| return | ||
| } | ||
| ref := treeRefName(e.ID).String() | ||
| if cur, ok := existing[ref]; !ok || cur != e.Tree { | ||
| out <- refUpdate{Ref: ref, Hash: e.Tree} | ||
| } | ||
| n := atomic.AddInt64(&processed, 1) | ||
| reportProgress(progress, n, int64(total)) | ||
| } |
| ref, err := repo.Reference(branch, true) | ||
| if err != nil { | ||
| // No v1 branch -> nothing to migrate. | ||
| return nil, nil //nolint:nilerr // absent branch is an empty list, not an error | ||
| } |
| // migrateRefsResult summarizes a run. | ||
| type migrateRefsResult struct { | ||
| Created int | ||
| Skipped int | ||
| Total int | ||
| } |
Note
Medium Risk
The command mutates local git refs in bulk; mistakes could repoint many checkpoint refs, though dry-run, skip-if-correct logic, and hidden CLI limit exposure.
Overview
Adds a hidden
entire checkpoint migrate-refssubcommand that backfills per-checkpoint git refs atrefs/entire/checkpoints/<ab>/<rest>/tree, each pointing at that checkpoint’s metadata subtree hash onentire/checkpoints/v1.The migration walks v1 (or reuses a TSV cache under the git common dir), compares against a snapshot of existing tree refs in parallel, skips refs that already match, and applies creates/updates via batched
git update-ref --stdin. Flags cover--workers,--cache-file,--refresh, and--dry-run; re-runs are idempotent.Unit tests cover the full pipeline; optional local benchmarks compare resolving a subtree by v1 tree walk vs direct tree ref (warm/cold).
Reviewed by Cursor Bugbot for commit 06be3c7. Configure here.