docs: propose the S3 versioning design (per-key version tree)

[frrist]

The implementable spec for S3 bucket versioning, built on the per-key version-tree leaf proposed in the PR #2 architecture review (@hannahhoward's design-comment), which supersedes architecture.md §3's composite-key invertedVersionId encoding.

Covers the data model (ObjectLeaf / prev sub-MST keyed by inverted seq), the token grammar, the write rule per versioning state, read/delete/list semantics pinned against the versitygw conformance suite, reference-index interplay, and a file-by-file implementation map.

Design doc only — the implementation lands in a follow-up PR once this is ratified. (The design has been validated end-to-end by a prototype in this worktree: unit suite plus the upstream versioning conformance categories against the live smelt stack.)

[alanshaw]

I think I need to understand properly how this all works before I read any more. I'm not sure if I'm missing something?

[alanshaw]

Ewww

[frrist]

Agreed, and it's AWS's rule: S3 fixes the null version's id to the literal string null, and the conformance suite asserts the exact echo (Versioning_PutObject_suspended_null_versionId_obj). The consolation in this design is containment: the null version costs one string compare plus one uint64 on the leaf (NullSeq), where the composite-key scheme let it leak into the key encoding.

[alanshaw]

Can we just inline this in the current node? Does it need to be separate?

[frrist]

Inlining the manifest into the leaf would break invariant 5 (a version's identity never changes). The manifest CID is the version's stable handle: an overwrite pushes that same CID into the prev tree, promotion pulls it back, and nothing is rewritten. The reference index (§8) counts per-version claims on the strength of that stability. Inlined, a displaced manifest would need re-serializing as its own block, which mints a new CID, and the full blob list would sit on a block that every read and list fetches. §2.1 now spells this out (606460e).

[alanshaw]

Below we say that Version ID comes from the manifest - why is it then repeated here?

[frrist]

Good catch. That sentence was meant to be scoped to prev-tree entries (their MST value is a bare manifest CID, so for them the id lives only in the manifest), and as written it contradicted this field. The copy here is deliberate: it lets the leaf answer questions without fetching a manifest. The write rule decides retain-or-discard from displaced.VersionID (§5.2), and resolution short-circuits on Current.VersionID / Current.Seq (§6.1). Manifests are immutable, so the copy can't drift. 606460e rescopes §2.2 and adds the rationale to §2.1.

[alanshaw]

Hmm, I'm not clear on how this works - how do you get a particular version if the version ID is not in the key?

[frrist]

The token carries its own position: a numbered version id is a ULID whose low 64 entropy bits are the version's seq (§3). A version-scoped read parses the token, checks Current.Seq, and otherwise seeks prev.Get(revSeqKey(seq)) directly, O(log n), no scan. It then fetches the manifest and confirms the stored VersionID equals the full token; any well-formed ULID parses, so without that check a token this bucket never minted could land on a real seq and serve a version the caller never named (§6.1). versionId=null is the one token with no position inside it, which is what the leaf's NullSeq field is for. The doc buried all of this in §3 and §6: 606460e adds an end-to-end walkthrough at the top of §2 so the mechanism lands before the data structures.

[alanshaw]

These are both repeated in VersionNode?

[frrist]

Yes, and each side has a job. The manifest is the authoritative, self-contained record: prev entries are bare manifest CIDs (§2.2), so for a noncurrent version the manifest is the only place its id exists. List rendering reads it there, the §6.1 check verifies against it, and promotion (§7.2) rebuilds Current from it. The VersionNode copies exist so leaf-only decisions skip the manifest fetch (see the thread on line 51). Manifests are immutable, so the two can't drift: the copy is a cache, and the manifest stays the record. §2.1 and §2.3 now say this in the doc (606460e).

[frrist]

Fair, and the doc's ordering caused it: the data structures (§2) landed before the mechanism that motivates them (§3's seq-in-the-token, §6.1's resolution). 606460e restructures:

• §2.0 (new): an end-to-end lookup walkthrough, including the token-verify guard, so the mechanism comes first.
• §2.1: why VersionNode caches two manifest fields, and why the manifest stays a separate block.
• §2.2 / §2.3: the 'id lives in the manifest' claim is now scoped to prev-tree entries, which is all it ever meant.

The short version: the top MST still maps plain object keys, now to a small per-key leaf. The leaf holds the current version inline, so plain reads and lists cost one extra block and never touch history. Old versions live in a per-key sub-MST keyed newest-first by seq. A version id is a ULID with the seq embedded, so version-scoped reads seek straight to their target. Happy to walk through it live if that's faster.

[alanshaw]

Just checking this isn't refering to Piri and removing a blob only when it is removed from all spaces it is allocated in. This is Ingot tracking blobs and actually performing /blob/remove only when no objects (or versions of an object) in the bucket refer to it?

[frrist]

Yup, that's right. There are two layers of refcounting:

1. ingot's blob_refs counts claims per (space, digest) across all objects and versions in the bucket, and only at zero does ingot invoke /blob/remove that releases the space's claim on Piri.
2. piri's own cross-space accounting before physically deleting bytes (here).

I'll add a clairifcation that RemoveBlob in ingot means: "release this space's claim via /blob/remove and not physically delete".

[alanshaw]

and it would put the full blob list on a block that every read and list fetches.

You need it anyway for those responses. You cannot list objects in a bucket with just the data that is defined in the ObjectLeaf.

[frrist]

I think this design should obviate this? https://github.com/fil-forge/ingot/pull/49/changes#r3730655300

[alanshaw]

An inlined manifest would have to be re-serialized as a standalone block when displaced, which mints a new CID

I'm just saying for every key read we've gone from 1 block lookup to 2.

i.e.key -> ObjectManifest to key -> ObjectLeaf -> ObjectManifest for ALL buckets and all keys versioned or not. Further, I imagine the majority will not be versioned.

"minting" a CID is deterministic, so unless we're going to mutate the manifest, does it matter if we store a manifest block when we "revise" (aka displace) an object?

[frrist]

Yeah, fair and I think you've already solved this below via the discriminator suggestion. So we'll adopted both together:

• the top-MST value is now a union.
• A key holding exactly one version stores the bare ObjectManifest CID so one block per read
• It upgrades to an ObjectLeaf on its first revision supersession(?), keeping the leaf from then on.
• Unversioned buckets (the majority case, as you say) never pay the indirection; only keys that actually accumulate versions do, and those need the leaf hop anyway to route the seq lookup.

The price is a two-case decode at each read/write site, which I think is worth it.

[alanshaw]

We should use a version discriminator on our key values to allow upgrades to this format in the future without duck typing or re-writing all data and continue to support older versions.

e.g.

type ObjectLeaf union {
  | ObjectLeafV0 "/objectleaf/0"
  | ObjectLeafV1 "/objectleaf/1"
} representation keyed

type ObjectLeafV0 struct {
  current VersionNode
  prev Link
  nullSeq int
}

type ObjectLeafV1 struct {
   // completely different to v0
}

[frrist]

Adopted. Outline here: https://github.com/fil-forge/ingot/pull/49/changes#r3730655300

[alanshaw]

Can we use a different word to displacement? "Revision" maybe?

[frrist]

Renamed, though I went with "supersession"/"superseded" rather than "revision": "revised" reads ambiguously about which version it names (the new one or the pushed-down one), while "superseded" can only mean the old current — and architecture.md already uses "superseded" for exactly this. Happy to switch if you feel strongly about "revision".

[alanshaw]

Gotcha, sounds fine - it's better than displacement.

[alanshaw]

What does "promotion" mean? Is this just retrieving a version? We don't ever actually move previous versions back to current right?

[frrist]

We do move one back: deleting the current version by versionId makes the newest remaining prev entry current again — S3 semantics; removing the latest version re-exposes the next-newest. "Promotion" is that move, nothing else. I'll update the doc to make this clearer.

[alanshaw]

Ah yes of course, thanks.

[alanshaw]

Suggested change

	### 5.2 Displacement
	With `displaced` = the existing `leaf.Current` (if the key exists) and `new` = the incoming
	### 5.2 Revisioning
	With `revised` = the existing `leaf.Current` (if the key exists) and `new` = the incoming

[frrist]

Applied via the rename thread above, with "Supersession" in place of "Revisioning".

[alanshaw]

LGTM 🚀

[alanshaw]

Personally I'd also envelope this as well like { "/objectmanifest/0": <ObjectManifest fields> } so you gain the ability to upgrade the format of a manifest over time as well.

You can also keep your manifest/leaf types separate like:

type Value struct {
  Manifest   *ObjectManifest   `cborgen:"/objectmanifest/0"`
  ManifestV1 *ObjectManifestV1 `cborgen:"/objectmanifest/1"`
  Leaf       *ObjectLeaf       `cborgen:"/objectleaf/0"`
}

...or, leave it as is and add it when we need it.

[frrist]

Adopted in full via 43dfdf9. Every value block is now the keyed union, manifests under /objectmanifest/0 and leaves under /objectleaf/0, with the codec generated from essentially the struct you sketched.