Future storage: SQLite metadata + S3 immutable pack hybrid

[yifanxuaaa]

Goal

Design a future LayerFS hybrid deployment with SQLite as authoritative transactional metadata and S3-compatible object storage for immutable packs. Preserve the canonical filesystem model and bounded reads while allowing retained pack payloads to live outside the local SQLite database.

This is a focused follow-up to #52, not approval to implement a second backend during v0.1.4. It develops the object-storage option concretely; #52 still owns the broader cloud topology and release decision. Durability/fsync/recovery qualification belongs with #69.

Current priority and scope boundary

The owner has deferred SQLite allocation tuning: do not add page-size experiments or a schema-consolidation campaign to the current storage milestones. Continue the approved v0.1.4 work with all authoritative data, including pack BLOBs and locators, in SQLite and required encoding complete synchronously before public return.

This issue does not authorize changing those acknowledgements, adding background packing, migrating existing Stores, or starting cloud implementation now. Moving pack payloads to S3 is not a claim that the minimum allocation of SQLite metadata tables disappears.

Proposed mental model

Workspace / FUSE edits
    -> shared canonical construction: CAS + CDC + COW
    -> bounded physical encoding: FULL/DELTA records, RAW/compressed groups
    -> immutable S3 packs
    -> authoritative SQLite publication: locators + retained roots + conditional heads

Reader -> authorized metadata lookup -> selected remote pack ranges
       -> bounded decode / physical base reconstruction -> ObjectId authentication
       -> filesystem bytes

• Canonical ObjectIds and filesystem semantics do not depend on local paths, SQLite rowids, pack placement, or S3 keys.
• SQLite owns object locations, retained roots, Branch/LayerStack heads, and transactional publication. A service may own that SQLite database for multi-machine access; clients must not independently write replicated copies and expect them to merge.
• S3 owns immutable pack payloads. Do not overwrite an old pack for each file edit. Preserve direct COW range/structure reuse and exact CAS sharing.
• Init and Commit retain their distinct lifecycle semantics while sharing construction and physical storage machinery.
• Small and large files use the same canonical path. Do not introduce a separate small-file storage engine.

Decisions and acceptance checklist

Portable format and export

• Define a portable locator mapping from ObjectId/canonical length to an immutable remote pack reference and group/record location. Translate local numeric pack IDs; do not treat them as globally meaningful. Content-addressed pack names are optional unless justified.
• Demonstrate whether whole local pack BLOBs can be exported byte-for-byte with a translated manifest. State when selective export requires overfetch or repacking; do not promise universal zero-repack migration.
• Preserve explicit wire/codec versions and bounded encoded/decoded sizes. Treat location metadata as untrusted for content integrity: validate framing and authenticate reconstructed canonical objects.
• Enumerate both logical reachability and physical delta-base closure for export, transfer, retention, and eventual garbage collection.

Remote reads and placement

• Specify a range-read path that avoids mechanically translating the current header/directory/group local reads into three sequential remote requests. Evaluate portable group offsets, bounded directory caching, and contiguous range coalescing using the smallest sufficient mechanism.
• Account for cold metadata traversal, repeated group decoding, sequential reads, random reads, and cross-pack FULL-base fetches. Depth-one deltas bound reconstruction, not network latency.
• Decide whether current local pack sizes are suitable remotely or whether export-time aggregation is warranted. Keep compression-group bounds separate from remote object size; avoid premature pack-size changes in v0.1.4.
• Define bounded cache ownership and eviction, including encoded/decoded bytes and tenant boundaries. Do not add redundant caches or assume a warm cache in performance claims.

Publication, failures, and access

• Define local acknowledgement versus remotely available/durable publication. For shared remote publication, required packs must be available before publishing roots and conditionally advancing heads. S3 upload and SQLite transaction are not one atomic transaction.
• Specify idempotent upload/publication, lost replies, partial failures, and orphan-pack cleanup without exposing incomplete roots or deleting physical bases required by retained state. Coordinate durability/recovery requirements with v0.3.0: Durable Commit and explicit fsync/recovery guarantees #69 rather than redefining them implicitly here.
• Preserve applicable Branch reconciliation semantics. Replication does not reconcile independently writable SQLite databases.
• Define tenant/project authorization for metadata and pack access. ObjectId knowledge is not authorization, and global CAS identity does not automatically permit cross-project deduplication.

Suggested rollout

1. Design and export boundary: document the selected topology, portable manifest, dependency closure, compatibility, and acknowledgement contract. An export/import slice may establish pack portability before live remote FUSE reads are attempted.
2. Minimal hybrid implementation: one authoritative metadata owner, immutable remote packs, bounded reads, and explicit publication ordering. Reuse existing owners; add no speculative backend plugin hierarchy, distributed scheduler, sharding, or replicas.
3. Cloud verification under v0.3.0: Cloud native integration and S3-compatible persistence #52/v0.3.0: Durable Commit and explicit fsync/recovery guarantees #69: qualify actual remote behavior, failure handling, concurrency, and cost under a separately approved plan.

Future evidence must separate local disk, remote stored bytes, cache/spool bytes, request counts, transfer bytes, foreground latency, queue time, CPU/RSS, and retention overhead. Compare cold/warm reads and small/bulk operations. Local S3-compatible verification does not establish AWS latency or durability. Do not launch a new benchmark family or adopt a population/environment as part of creating this issue.

Completion criteria

The hybrid contract and its relationship to #52/#69 are explicit; the selected implementation preserves historical filesystem readability and canonical identity, publishes only complete states, bounds remote read/decode work, and has retained evidence for its storage/latency/cost tradeoff. No claims that SQLite can simply reside on shared object storage or that cloud placement alone improves total storage efficiency.

References

• Parent cloud intent: v0.3.0: Cloud native integration and S3-compatible persistence #52
• Durability and recovery scope: v0.3.0: Durable Commit and explicit fsync/recovery guarantees #69
• Local storage checkpoint: storage: M4.5 4-KiB creation and dual-layout compatibility checkpoint #81
• docs/roadmap/0.1/0.1.4/storage-architecture-spec.md
• docs/roadmap/0.1/0.1.4/sqlite-storage-format.md
• S3 GetObject and range restrictions
• S3 consistency model