feat(s3): implement bucket versioning (per-key version tree)

[frrist]

Implements the S3 versioning design of docs/s3-versioning.md — this PR is stacked on the design
branch of #49: review the spec (and the architecture.md alignment) there, the code here.
CompleteMultipartUpload commits through the versioning write rule on top of the deferred-accept
flow (#40), with the conditional-write preconditions re-checked under the commit lock.

What's in it

• Data model (§2): every top-MST value block is a cborgen keyed union
  (bucket.ValueUnion): a key holding one version stores its manifest under
  "/objectmanifest/0" — one block, one fetch — and gains a bucket.ObjectLeaf under
  "/objectleaf/0" (current version inline + per-key prev MST keyed newest-first by inverted
  seq) at its first retained supersession, keeping it from then on. Every manifest block carries
  its union key — as a value and as a prev-tree entry — so a version's identity CID names one
  encoding. The strict ObjectValue / EnvelopedManifest / EnvelopedLeaf wrappers require
  exactly one union arm, so a block with no known key (a newer format, or a pre-union bare
  block) is an error, never a zero-filled cross-type decode; either arm's format can be revised
  under a new key with old and new blocks coexisting. ObjectManifest carries Seq/VersionID.
• Version identity (§3, §5.1): per-bucket seq allocated under the bucket commit lock
  (registry.AllocVersionSeq); client ids are strict ULID tokens carrying the seq in the low 64
  entropy bits, "null" for unversioned/Suspended writes.
• Bucket state (§4): PutBucketVersioning / GetBucketVersioning over buckets.versioning.
• Write rule (§5): one commit helper (commitVersion) behind PUT / Copy / CompleteMPU /
  marker insertion — supersession per state, new keys and null-over-null writes staying bare,
  the single-null invariant with prev-null eviction, GC candidates for discarded manifests and
  superseded leaves. Response version ids gate on the state read under the lock, so a racing
  PutBucketVersioning can't mint an id the response omits.
• Reads (§6): resolveVersion — on a bare key every versionId class resolves against the
  single manifest; on a leaf key, current / null / token seek with the token-verify guard.
  Marker semantics 404/405 with x-amz-delete-marker + Last-Modified on the error response;
  NoSuchVersion errors carry <Key>/<VersionId>.
• Deletes (§7): numbered/null delete markers, version-scoped delete (a bare key's match
  drops the key; on a leaf, promotion of the newest survivor — a leaf is never downgraded,
  invariant 6), DeleteObjects dispatch (explicit DeleteMarker: false for non-marker version
  deletes), versioned DeleteBucket guard.
• Reference index (§8): per-version blob_refs claims; releases only on discard;
  null-replacing-null goes through the set-diff so unchanged digests never churn.
• Lists (§9): listWalk marker skip (one manifest fetch per bare key, leaf + manifest per
  superseded key); ListObjectVersions with combined-count pagination, marker-pair resumption,
  and delimiter grouping.
• Conformance partition (§11): itest/versity_versioning_test.go — PutBucketVersioning,
  GetBucketVersioning, ListObjectVersions, Versioning, and the never-versioned
  ListObjectVersionsVD listing group, curated into the pass/XFail ratchet. The versioned
  categories run with WithVersioningEnabled so upstream teardown empties buckets per-version.

Verification

• make build, go vet ./... and go vet -tags itest ./itest/, gofmt -l clean,
  go mod tidy -diff clean, and make gen is a no-op on the committed cbor_gen.go (the
  union codec is generated; only the thin strict wrappers are hand-written).
• Unit: GOWORK=off go test ./... -count=1 green — 18 versioning tests covering token
  classification, every §5.2 supersession row, first-supersession leaf creation, prev-null
  eviction, promotion (and that the leaf survives it), per-version claim add/release on the
  refindex harness, pagination, delimiter grouping, and the manifest-arm form of unversioned
  keys asserted through ObjectValue; 3 codec tests pinning both arms' round-trips and the
  rejection of unknown union keys and pre-union bare blocks.
• Itest (smelt stack, this tree's binary): the full suite runs green. With the delete-release
  chain in place (ingot /blob/remove from feat: deferred-accept multipart (FIL-520) + network blob removal on DeleteObject (FIL-588) #40, hilt#37's blob.Remove write-set grant,
  smelt#19's piri blob/release + blob/reject seed delegations), the versioning partition
  is 52 pass rows with a single XFail (DeleteObject_non_existing_objects, which needs
  object-lock-enabled bucket creation that ingot does not model).

Follow-ups (noted, out of scope)

• Superseded prev-tree/MST interior nodes are not recorded in gc_candidates, matching existing
  top-MST practice (spec §5.3 documents the policy).
• ListObjectVersions shares V1 listWalk's exact-fill IsTruncated quirk.
• Candidate refactors: extract a supersede() helper from commitVersion; have prevHead
  return the tree key so promotion uses the key-derived seq instead of the manifest copy.

Closes (together with #49) https://linear.app/filecoin-foundation/issue/FIL-585
🤖 Generated with Claude Code

[Copilot]

Pull request overview

Implements S3 bucket versioning using the per-key version-tree (“ObjectLeaf”) data model: object keys map either directly to a single manifest CID (bare form) or to an enveloped leaf that tracks current + prior versions via a per-key MST. This wires version-id handling (ULID tokens + "null"), version-scoped reads/deletes, delete markers, ListObjectVersions, and the registry/state + persistence needed to allocate per-bucket version ordinals.

Changes:

• Add versioning core logic in s3frontend (version-id grammar, commit write rule, version resolution, version-scoped delete, and ListObjectVersions).
• Extend registry + migrations to persist bucket versioning state and allocate per-bucket version sequences.
• Add unit + integration conformance coverage for versioning behaviors and pagination/teardown semantics.

Reviewed changes

Copilot reviewed 24 out of 26 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
s3frontend/version.go	Core versioning helpers: version-id parsing, commit write rule, resolveVersion, version-scoped delete helpers.
s3frontend/version_test.go	In-process unit tests covering versioning write rule, resolution, delete markers, deletes, and listing semantics.
s3frontend/object.go	Route PUT/GET/HEAD/DELETE (and listWalk) through version resolution/write rule; version-aware blob_refs reconciliation.
s3frontend/multipart.go	CompleteMultipartUpload commits through versioning write rule and returns version id when configured.
s3frontend/listversions.go	Implements ListObjectVersions over bare/leaf forms with combined-count pagination and delimiter grouping.
s3frontend/copy.go	CopyObject supports copy-source ?versionId, marker handling, and version-id response shaping.
s3frontend/conditions.go	Precondition helper updated to treat delete-marker current as “no object” and to use version resolution.
s3frontend/bucket.go	Implements Get/PutBucketVersioning and versioned DeleteBucket not-empty error mapping.
s3frontend/backend.go	Updates unsupported-ops comment to reflect newly implemented versioning.
registry/stores.go	Clarifies meaning of NullVersionID now that per-version ids exist.
registry/registry.go	Adds bucket VersioningState, persisted in State, plus Registry APIs SetVersioning/AllocVersionSeq.
registry/postgres.go	Persists bucket versioning state + sequence allocation in Postgres implementation.
migrations/sql/00003_stores.sql	Documents schema semantics for version_id, buckets.versioning, and next_version_seq.
itest/versity_versioning_test.go	Adds curated upstream conformance tables for versioning categories (pass + XFail).
itest/versity_test.go	Runs versioning categories with a versioned S3Conf so upstream teardown deletes per-version.
itest/stack_test.go	Adds forgeS3ConfVersioned to enable upstream versioned teardown mode.
itest/README.md	Updates docs to include versioning conformance file.
inmem/store.go	Adds in-memory tracking of bucket versioning state and per-bucket version seq allocation.
go.mod	Adds github.com/oklog/ulid/v2 dependency explicitly.
go.sum	Updates ULID v2 sums and related entries.
gen/main.go	Adds bucket types to cborgen list to keep generated codecs in sync.
CLAUDE.md	Updates testing documentation to include versioning conformance file.
bucket/manifest.go	Adds Seq/VersionID fields and updates delete-marker semantics documentation.
bucket/leaf.go	Introduces VersionNode/ObjectLeaf, envelope codec, and exact discriminated decoding via ObjectValue.
bucket/leaf_test.go	Tests envelope round-trip, bare manifest decode, and unknown-envelope rejection behavior.
bucket/cbor_gen.go	Updates generated CBOR codecs for new/changed bucket types and fields.

Files not reviewed (1)

• bucket/cbor_gen.go: Generated file

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[alanshaw]

🚀

[alanshaw]

Why is this "Nocache" but when walking the versions below we do not use "Nocache"?

[frrist]

The outer walk is over the whole bucket, and it can touch way more keys than it returns. With a delimiter, a prefix hiding a million keys counts as just one result — but the walk still steps over all million leaves to get past it. If it kept every node it loaded, one List request could hold a huge slice of the bucket's tree in memory. So: Nocache.

The inner walk is over one key's versions (sub MST), and every leaf it touches counts toward MaxKeys. So it can never touch more than ~1000 leaves before stopping. On top of that, the little tree object (pt) is created inside the callback and garbage-collected the moment that key is done. It can't accumulate anything, so Nocache wouldn't change anything.