Architecture migration (Phases 1–6): object-aligned blobs, sync durability, dedup/reference-counted delete, S3 correctness + multipart by frrist · Pull Request #9 · fil-forge/ingot

frrist · 2026-06-24T00:41:55Z

Ingot Forge-Mode: Status & What's Left

Where we stand (one paragraph)

Ingot's storage/delete/retrieval core is built and validated over real Forge (sprue + piri + indexing-service in smelt). Single-object PUT/GET/HEAD/DELETE, ranged reads, conditional requests, checksums, copy, multi-delete, and multipart all work end-to-end over the network. This PR brings ingot in line with the target docs/architecture.md, validated forge mode in smelt for the first time, found+fixed one real forge bug (dedup 500), built the appliance read tier (read blobs back from piri via a local location table — no indexer needed for reads), and turned the smelt e2e into a real per-case regression gate covering single-object and multipart. What remains is delete finality (the remove/unallocate network primitives, which are stubbed because the piri-side handlers don't exist yet), crash recovery, and a few unimplemented S3 multipart sub-APIs.

1. Ingot — this PR (`feat/arch-migration-phase1`, 22 commits ahead of `main`)

Migration Phases 1–6 (22e926e→1ef6fd5): object-aligned blobs, spool + synchronous per-blob upload, single-plane catalog log, reference-counted delete, conditional requests, checksums, DeleteObjects, CopyObject, multipart. Plus b5eb52e (3 review-found bug fixes).

S3-surface batch: 85a74cb HEAD-Range + Expires; 1cdaaf5 GET/HEAD ?partNumber=N; c047a11 server-default CRC64NVME checksum.

Phase 7 (forge hardening, smelt-validated):

14a22d1 — forge dedup fix: uploadBlobs skips re-uploading a blob already in blob_locations (was causing 500s on re-PUTs over forge).
55bb542 — per-case capture: ingottest.Run recovers per-case pass/fail names so the smelt gate can compare sets.
d86e432 — registry.LocalLocator: the appliance read tier — resolves blob locations from the local blob_locations table and drops into the existing Forge.GetBlock retrieval path; module.go wires it instead of the indexer.

2. Smelt — validation harness (`fil-forge/smelt` PR #10, draft — depends on this PR)

The ingot-in-smelt integration lives there: systems/ingot/ (compose + forge-mode config; a postgres:16-alpine; migrations at startup on a fresh DB), the SMELT_WORKSPACE=1 workspace build (rebuilds ./cmd/ingot from local source and bind-mounts it over the image), and pkg/stack support (IngotEndpoint/Exec/Logs).

Three e2e tests (the validation surface):

TestIngotNativeProvision — ingot self-provisions its space (guppy-free) + Smoke suite over forge.
TestIngotVersityConformance — the gate: runs Smoke + Multipart against an in-process baseline and forge, asserts forge failures ⊆ baseline (so any forge-specific regression fails by name; out-of-scope cases self-calibrate out). Last run: 228 cases, 0 forge-specific failures.
TestIngotForgeReadAfterEviction — PUT → wipe /data/spool → GET re-fetches from piri via LocalLocator. 512 KiB round-tripped.

3. What's still stubbed/deferred in ingot (forge-mode glue)

Item	State	Detail
`remove(digest)`	logged no-op (`uploader/blob.go:95`)	Reference-index bookkeeping (count→0 ⇒ call site) is fully built + tested; the network release is stubbed because the piri handler doesn't exist. Bytes accumulate on piri.
`unallocate(digest)`	does not exist	No client method at all; same piri-handler blocker.
Crash recovery	deferred	No `upload_intents × blob_refs` reconciliation job. A crash between commit and reconcile leaks (never loses) — §12 known boundary.
True part-parking	not done (optional)	Current flow spools parts at `UploadPart`, uploads+accepts at `Complete` (validated). "Upload-early + accept-at-Complete + unallocate-on-abort" is a latency refinement, not correctness.
`UploadPartCopy`, `ListMultipartUploads`	`ErrNotImplemented`	Inherited from `backend.BackendUnsupported`.
`ListParts` (S3 API)	partial	The internal store method is used by `Complete`; the S3 listing API isn't fully surfaced.
Config validation lag	cleanup	`cmd/config.go` still requires `indexer_endpoint`/`indexer_did` in forge mode even though `LocalLocator` reads make the indexer unused — should become optional.
Versioning, Lifecycle/`Expires`-expiry	deferred by scope	Per architecture §12.

4. Changes needed in Piri / Sprue

The delete path is the real external blocker. libforge has the /blob/remove binding (commands/blob/remove.go); what's missing is the server side:

Need	Service	Blocking?	Why
`/blob/remove` handler (per-space claim release; physical delete + piece-retire at zero global claims)	Piri + Sprue	BLOCKING for delete finality & GC	Ingot's `RemoveBlob` call site is built; it no-ops until this lands. Piri must keep per-`(digest,space)` allocation rows to count claims.
`/blob/unallocate` handler (drop a parked blob)	Piri + Sprue	BLOCKING for true-parking abort	Only needed once parts are uploaded-early at `UploadPart`; with the current accept-at-Complete flow, Abort just drops local spool (no forge state), so this isn't blocking today.
Ingot-timed accept (separable park-vs-conclude so accept can be deferred)	Sprue (+ Ingot)	needed for true parking	Today accept fires on the client's conclude; ingot doesn't issue accept (piri requires the upload-service DID).
De-dup at accept (don't re-aggregate a digest that's already a live piece)	Piri	OPTIONAL now	This session's client-side dedup backstops it — ingot owns one space, so re-adds are always within-space dups and caught before `/blob/add`.
Configurable `min`/`max`, compaction (Regime B), complete the on-chain delete signature	Piri	OPTIONAL / perf	`MinAggregateSize` hardcoded 128 MiB; whole-root delete wired but `extraData` signature incomplete. Aggregation is entirely piri-side; ingot just calls `remove`.
Parked-allocation GC + honor `Expires`	Piri	OPTIONAL	Bounds leakage if an abort never arrives.
Streaming commP (lift 256 MiB ceiling), batch allocate/accept, `allocate-by-size`	Piri	OPTIONAL / future	Throughput/RPC optimizations.

5. Changes needed in the indexing-service — mostly obviated

This work removed the indexer from ingot's read path. The appliance topology (R0/R1) now reads blob locations from ingot's local blob_locations Postgres table via LocalLocator, validated by the eviction e2e. Consequences:

Reads: the indexing-service is no longer needed for ingot reads in the appliance topology. (The indexer-backed locator stays available behind the seam for the full-vision topology — ForgeConfig.Locator is just injectable.)
One remaining indexer item — and it's optional here: indexer delete/retraction by (space, digest) (IPNI removal when the last claim drops) is still to-build, but it only matters when ingot publishes to and reads from the indexer. In the appliance topology it doesn't, so this is not on the appliance critical path — it's full-vision cleanup.
Net: for the appliance, the indexing-service is effectively out of ingot's loop (piri still emits a location commitment at accept as a side effect, but ingot neither requires nor reads it). The only ingot-side cleanup is dropping the indexer-required config validation (item in §3).

6. Cross-repo status & next steps

Smelt validation harness is up as draft PR #10 — held in draft until this PR merges and a new ghcr.io/fil-forge/ingot:dev image publishes (its forge e2e exercise the LocalLocator + dedup behavior introduced here; they pass today only via SMELT_WORKSPACE=1 local builds).
Highest-value remaining ingot work that is not blocked on piri: crash recovery (the upload_intents × blob_refs reconciliation job).
The one cross-repo dependency gating delete finality: the piri /blob/remove handler (libforge binding exists; the server handler is to-build). Everything physical-delete is no-op until it lands.

🤖 Generated with Claude Code

alanshaw

Sorry I didn't get far through this and have to step out for a bit. Will circle back.

alanshaw · 2026-06-26T12:55:32Z

+	// — the appliance read tier (e.g. registry.LocalLocator over blob_locations).
+	// When nil, an indexing-service-backed locator is built from
+	// IndexerEndpoint/IndexerDID. Either way the retrieval path is identical.
+	Locator locator.Locator


Ooh nice 👌

alanshaw · 2026-06-26T12:59:24Z

+		if !errors.Is(err, ErrNotFound) {
+			return nil, err
+		}
+	}


Might be worth implementing as a "tiered" blockstore that just iterates over a list of BlockGetters until one doesn't return an error.

alanshaw · 2026-06-26T13:05:48Z

+// PutBlock writes a raw block to the spool, keyed by its multihash. Writing is
+// atomic (write to a temp file, then rename) so a crash mid-write never leaves a
+// partial blob readable under its digest.
+func (s *Spool) PutBlock(_ context.Context, blk block.Block) error {


Wait, passing in a block.Block here implies that the data has been received and hashed and is presumably stored in...memory? We should write directly to disk calculate the hash as we're doing so. Otherwise the appliance is going to have a huge RAM requirement as well as a big spooling disk.

Fixed in 223b557, and I just cleaned up the leftover in be552e3. Write path now streams to disk with the hash inline, Read path also streams.

alanshaw · 2026-06-26T13:07:21Z

+// is expected and cheap: it lets the layered read path fall through to the log
+// (for catalog blocks, which are never spooled) or the network tier (for a body
+// blob that has been evicted).
+func (s *Spool) GetBlock(_ context.Context, c cid.Cid) (block.Block, error) {


Again, these are data blobs right? So up to 5GB in size. We're not going to be able to read many of these concurrently if we're buffering everything into memory.

Peeja · 2026-06-30T21:22:27Z

+// CopyObject copies an object as a metadata-only operation under dedup: it
+// resolves the source manifest and writes a new destination manifest pinning the
+// SAME body blobs (same digests), adding a reference-index claim per digest. No
+// bytes move and no Forge upload happens — the blobs already exist. Honors


Er, hang on. Is this supposed to this work across tenants? The object is encrypted for a tenant, with their key. The blobs won't make any sense in someone else's bucket.

On the cross-tenant angle specifically the FEE RFC lists server-side copy as a deferred non-goal, and since hilt is still being built there isn't a tenants API to account for/use. So there's no settled answer to bump up against here yet, yeah?

fwiw I (re)read through the RFC and I don't see why metadata-only copy would break within a domain/space: reads unwrap via the region wrap (per region×blob, not per-tenant), so a shared blob still decrypts for the destination — the only tenant-specific bit is the recovery wrap + kid baked into the envelope. So even across tenants I'd expect the read itself to work, with just the recovery/ownership wrap being the thing that needs re-doing. But I'd prefer to default to your judgement here as you're closer to the encryption work. Does this track? Or is there something about the per-tenant keying that makes a shared blob genuinely unreadable in another tenant's bucket?

I opened #16 with a few more details on this.

Oh, if this is going to change during R0 → R1, that's fine. I just figured it would be easier to build the full copy now than the reference counting, but maybe it's not.

Some of this is an open question, since we don't have a path planned that actually uses the tenant keys for reading. But I'd expect that if we were to provide such a path, we'd disallow using some other tenant's private keys to decrypt the first tenant's data, even if that's what the header points at. I guess it wouldn't be terrible to do that, since the reading tenant owns the blob that's being decrypted? But it feels pretty odd to me. I'd be more comfortable with each tenant's blobs encrypted with their own key.

Notably, there's still a bit of a shortcut here for "re-encrypting": we only need to re-wrap the CEK for the new tenant, change the header, and reuse the ciphertext. But that's still a new blob that needs to be added, it just saves some compute work.

Peeja · 2026-06-30T21:22:58Z

+// bytes move and no Forge upload happens — the blobs already exist. Honors
+// MetadataDirective (COPY = inherit source metadata; REPLACE = take it from the
+// request) and the x-amz-copy-source-if-* preconditions, and supports a
+// cross-bucket source in the same space.


Can you explain more? I'm not sure what a "cross-bucket source in the same space" is.

humm, yeah I expect this to change as encryption and tenant features/API land, but right now Ingot is getting ahead of that work.

So to clarify: At present, ingot runs single-region / single-space / single-tenant (one instance = one space, per DESIGN_NOTES.md (this will ofc change as we build this out)), and every bucket lives in that one space. So a copy's source and destination buckets are always in the same "domain". That's what makes copy metadata-only: I write a new destination manifest pinning the same body-blob digests and add a blob_refs[(space, digest)] claim — no bytes move, no re-upload. The blob is shared and reference-counted, so deleting the source doesn't touch the destination (same independent-object lifecycle S3 guarantees). I can extend this comment if we want to mention this, but right now it reflects the current state of the implementation.

So "cross-bucket source in the same space" means copy an object from bucket A to bucket B, where both bucket A and B are in the same space/region/tenant, which will always be the case given the current design and lack of tenant features.

Peeja

TBH, this is a lot to think through, and it seems to live in a slightly different place from what I've been working on. Same timeline, just an earlier point. I think it all lines up and makes sense to me, and I'm pretty sure there's nothing that could be devastatingly wrong and hard to adjust, so I'm in for a ✅. Let's keep this thing moving.

…igration phase 1) Bring the object body model in line with docs/architecture.md: an object's Body is now an ordered, contiguous list of content-addressed blobs (BlobRef{Digest,Offset,Length}) covering [0,size), instead of a single Content CID over a fine-chunked FixedChunkerIndex DAG. - bucket/manifest.go: Body -> {Size,SHA256,MD5,Blobs,IndexRoot}; new BlobRef; drop Content/Format + FixedChunkerIndex. ObjectManifest gains a stored ETag (verbatim; needed for multipart later) and a reserved DeleteMarker. versionId is intentionally not stored (it is a function of the MST key); IndexRoot is reserved (nullable) for a future multi-shard credible-exit record. - bucket/chunker.go: FixedChunker -> BlobSplitter, coarse-split at MaxBlobSize (default 256 MiB) into one raw block per blob; reader iterates Body.Blobs by offset/length. Fixes a latent buffer-reuse corruption bug (the old loop reused one buf across chunks while the staging store held them by reference) that broke any multi-chunk object; never surfaced because tests used sub-1MiB single-chunk bodies. - s3frontend: store ETag at PutObject; etagOf reads it back. - config/server/testing: ChunkSize -> MaxBlobSize (WithMaxBlobSize). - testing/blobsplit_test.go: multi-blob (boundary-crossing ranged GETs) and zero-byte round-trips. GetObject/large_object promoted xfail->pass (the multi-chunk fix makes it pass). - docs/IMPLEMENTATION_PLAN.md: phased migration tracker. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…(arch migration phase 2) Stand up the relational surface the upload/storage/delete architecture needs (docs/architecture.md §5-§7, Appendix C), as pure additive surface with no production callers yet — the in-memory suite stays green. - migrations/00003_stores.sql: blob_refs (reverse reference index), upload_intents (local-store lifecycle), blob_locations (the appliance local location table, in place of the indexing-service for body reads), multipart_sessions/parts (single-winner latch + ordered part blobs), gc_candidates; buckets gains space (DEFAULT '') + reserved versioning / next_version_seq + buckets_space_idx. - registry/stores.go: focused interfaces (BlobRefStore, IntentStore, LocationStore, MultipartStore, GCStore), row types, state constants, NullVersionID sentinel. stores_postgres.go implements them (jsonb metadata, bytea[] digests, ON CONFLICT upserts, latch via RowsAffected). State gains Space (read-only for now; Create still defaults it to ''). - inmem: MemStore mirrors all five stores with deep-copy semantics. - Tests: inmem unit tests (ref-count-to-zero, intent state machine, ordered parts + cascade, latch single-winner under -race). DSN-gated live tests (up_live_test.go, postgres_live_test.go) validate the migration + every store's SQL against a real Postgres. The blob_refs PK (digest,bucket,object_key,version_id) excludes space by design: space is denormalized and a (bucket,key,version) belongs to one space. buckets.space population is wired in phase 3. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…arch phase 3) Squashes the phase-3 sub-steps (+ its implementation-plan tracking): - 3a/3b: local blob spool + synchronous per-blob upload to Forge - 3c: delete the dead data PlaneLog (catalog is the only plane) - 3d: conditional forge_root advance + blob-location recording Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…elete (arch migration phase 4) Make overwrite-in-place and DeleteObject safe under dedup by maintaining blob_refs and releasing a blob only when its last reference drops (docs/architecture.md §5, §6). - s3frontend.reconcileClaims diffs the prior vs new body-digest sets on every commit: a newly-referenced digest gains a claim; one no longer referenced loses its claim and, at zero (space,digest) claims, is queued for release; a digest in BOTH is untouched (a re-PUT of identical bytes, or a split object sharing blobs, never churns the row). Wired into PutObject (overwrite loads the prior manifest's digests) and DeleteObject. - uploader.BlobRemover seam: RemoveBlob(digest) runs AFTER the commit, off the critical section (a 200 isn't gated on it). Forge.RemoveBlob is a logged no-op — libforge has the blob.Remove binding but the Piri/Sprue handler is to-build (§9) — so the bookkeeping runs end-to-end without the network primitive; bytes leak on Piri until the handler lands. - gc_candidates records the superseded manifest CID on overwrite/delete (write-only; no collector yet). Precise superseded-MST-node tracking deferred. - Wired BlobRefs/GC/Remover through ServerDeps/fx/harness/standalone. - s3frontend/refindex_test.go (white-box): dedup across keys, overwrite same vs different content, delete-to-zero, delete-one-of-two — each asserting blob_refs counts and the exact RemoveBlob calls. Claim updates run in the commit critical section (safe side: a lost cross-process CASRoot leaks rather than loses data); Phase 7 crash recovery reconciles upload_intents x blob_refs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…, copy, checksums (arch phase 5) Squashes: - 5a: conditional requests (If-Match/If-None-Match/If-(Un)Modified-Since), re-checked at commit - 5b/5c: DeleteObjects (batch) + metadata-only CopyObject under dedup - 5d: additional checksums (x-amz-checksum-*) validate + echo Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…arch phase 6) - Create/UploadPart/Complete/Abort with the single-winner session latch and the -N ETag - Fixes 3 correctness bugs found by adversarial review of phases 1-6: post-commit blob_refs reconcile (no divergence on commit failure), Complete latch reverts to open on a pre-commit error (stays abortable), deduplicated digest release (no double-free of a split-shared blob) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…; add §12 status & postponed items The body remains the north-star target. Two small inline tweaks flip the "target vs bootstrap MVP" framing now that the core is built (the §1 intro and the §4 manifest-diagram caption), and a new §12 records what is implemented in-process vs deferred: - scope decisions (versioning deferred → "null" sentinel + reserved fields; next-to-Piri so digest-before-upload kept; no Ingot-side aggregation; local blob_locations table behind a Locator seam instead of the indexer), - forge-mode glue validated in smelt (no-op remove/unallocate, the local-table Locator read tier, multipart true-parking, spool crash recovery, UploadPartCopy/ListParts), - the known reference-index ↔ commit atomicity boundary. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…lt checksum - HEAD honors Range (206/Content-Range, 416 on unsatisfiable); carry the Expires header - GET/HEAD by ?partNumber=N (part byte-span + x-amz-mp-parts-count) - server-computed default CRC64NVME checksum when the client names none Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

- fix: skip re-uploading already-located blobs (forge dedup; was a 500 on re-PUT) - registry.LocalLocator: local-table read tier for the appliance topology (no indexer) - stream body blobs to disk with inline hashing — never held whole in RAM on read or write - drop unused Spool methods (PutBlock/Has/Remove); trim to Path/WriteBlob/OpenBlob/GetBlock - test: recover per-case names from the conformance suite Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

frrist self-assigned this Jun 24, 2026

This was referenced Jun 25, 2026

Add the ingot system + forge-mode e2e validation fil-forge/smelt#10

Merged

test(smoke): drop order-dependent ListBuckets cases that flake under -shuffle #7

Closed

frrist requested review from Peeja, alanshaw, bajtos and pyropy June 25, 2026 20:33

alanshaw reviewed Jun 26, 2026

View reviewed changes

Peeja reviewed Jun 30, 2026

View reviewed changes

Peeja mentioned this pull request Jul 1, 2026

FEE wrap material on blob_locations (FIL-480) #15

Open

frrist mentioned this pull request Jul 1, 2026

CopyObject × FEE encryption: constraints to feed into FIL-273 before server-side copy is built #16

Open

4 tasks

Peeja approved these changes Jul 1, 2026

View reviewed changes

frrist mentioned this pull request Jul 1, 2026

[Tracking] Ingot forge-mode: remaining work after arch migration (#9) #17

Open

5 tasks

frrist and others added 9 commits July 1, 2026 16:06

frrist force-pushed the feat/arch-migration-phase1 branch from be552e3 to a59b01d Compare July 1, 2026 23:21

frrist merged commit 3d30e24 into main Jul 2, 2026
4 checks passed

Uh oh!

Conversation

frrist commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Ingot Forge-Mode: Status & What's Left

Where we stand (one paragraph)

1. Ingot — this PR (feat/arch-migration-phase1, 22 commits ahead of main)

2. Smelt — validation harness (fil-forge/smelt PR #10, draft — depends on this PR)

3. What's still stubbed/deferred in ingot (forge-mode glue)

4. Changes needed in Piri / Sprue

5. Changes needed in the indexing-service — mostly obviated

6. Cross-repo status & next steps

Uh oh!

alanshaw left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Peeja left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

frrist commented Jun 24, 2026 •

edited

Loading

1. Ingot — this PR (`feat/arch-migration-phase1`, 22 commits ahead of `main`)

2. Smelt — validation harness (`fil-forge/smelt` PR #10, draft — depends on this PR)