Repository navigation
Conversation
`with-storage-lock` is what keeps stratum's standalone `gc!` from sweeping chunks a concurrent `sync!` is about to reference. It allocated its monitor in a ConcurrentHashMap keyed by the store REFERENCE, on the stated reasoning that "File-store / S3-store records with content-based equality: two instances configured against the same backing storage share a lock — which is what we want (they protect the same physical data)." They do not. Two `connect-fs-store` calls on one path are neither `identical?` nor `=`, and hash differently: identical? false = false same hash? false So a process that connected twice to one store got two independent monitors and no exclusion at all — measured at 0.16ms of contention where the lock should have blocked for 300ms. The companion claim, that "under-locking is impossible: distinct keys always get distinct locks", had it exactly backwards. Two distinct keys for ONE physical store IS under-locking. The safe direction is the opposite, and it is the same rule `konserve.gc-guard` follows for the safe point: the key may be COARSER than the physical store — two stores collapsing onto one monitor merely over-locks — but never FINER. konserve's `:id` is coarser: a logical identity shared by replicas across machines, and within one JVM (all a monitor can reach) one id names one store. Stores built through a backend constructor carry no id and fall back to the object, which is the old behaviour; such callers should still avoid connecting twice. The regression test observes the monitor through the behaviour it exists for rather than reaching into private state: a lock held on one connection must exclude the other. It fails on the old key. Signed-off-by: Christian Weilbach <christian@weilbach.name>
…unify it Companion to the lock fix. Records that stratum's GC design follows from sharing datahike's store rather than owning its own, that both patterns already exist in the stack, and that the open question is store ownership rather than collection mechanism. Signed-off-by: Christian Weilbach <christian@weilbach.name>
Carries `konserve.protocols/store-id`, which the GC/sync lock now keys on (replikativ/konserve#159). Both pins bumped — the main dependency and the `:release` override. The `:local` alias stays for co-development but is no longer needed to build. 1430 tests / 5866 assertions against the published artifact. Signed-off-by: Christian Weilbach <christian@weilbach.name>
`stratum.warm/warm!` warms every indexed column of a dataset in one call: each column contributes its PSS tree and all of them share ONE budget round-robin (persistent-sorted-set's `warm-trees!`, new in 0.5.142), so no column can spend the budget before another gets any — which column the next query needs is not knowable at warm time. `warm-column!` scopes to one. Depth is a bigger lever here than in a thin-leaf tree, because a stratum leaf is FAT: each ChunkEntry carries its whole PersistentColChunk, so restoring the leaves IS materializing the column data. `:depth :interior` (the default) warms the spine — branches carry the ChunkStats measures, so stats and measure-driven pruning answer without touching a chunk — and `:with-leaves` with a sized `:budget` prefetches the data a known scan will read. The tests' oracle is CachedStorage's own miss counter, with one boundary learned by measuring rather than assuming: stratum's blob granularity is COARSER than its node structure (a whole small tree can be one or two blobs), so which blob a scan touches is a storage-layout fact and deliberately not asserted. What is asserted is the warm's contract — a full warm makes the scan that follows cost ZERO further reads, the budget is a hard ceiling (exactly N, not one more) split evenly across columns, :interior stays a small fraction of :with-leaves, and warming never makes a scan more expensive. Full suite on 0.5.142: 1435 tests, 5884 assertions, 0 failures. Signed-off-by: Christian Weilbach <ch_weil@topiq.es>
doc/warming.md is the user-facing story: the problem (a lazy-loaded dataset pays misses x RTT with nothing overlapping), the two entry points, why depth is a bigger lever here than in a thin-leaf tree (a stratum leaf carries its chunks, so :with-leaves IS materializing the column), the report as an early metric (:budget-exhausted? shows a decaying warm before p99 does), and — stated as prominently as what warming does — what it does not promise: results never change, :fetched counts restores not misses, and blob granularity is stratum's rather than the tree's. Linked from the README with a short section in the reading path between the quick start and durability. doc/gc-and-store-ownership.md was a development artefact — "written while landing gc-guard", present-tense drift like "landed this week" — describing its cross-project relationships more than any decision a reader could act on. The content deserves better: it records WHY stratum's GC differs structurally (shared store => contribute marks, owned store => sweep) and an ecosystem decision that is still open. Reframed as reference documentation and linked from the README's Ecosystem section, which is where a reader wondering about the shared store actually is. Signed-off-by: Christian Weilbach <ch_weil@topiq.es>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First consumer of persistent-sorted-set#25's warm push-down, and the reason the multi-tree budget round-robin exists:
stratum.warm/warm!warms every indexed column of a dataset in one call, all columns sharing one budget so none starves another;warm-column!scopes to one. Report is PSS's,:by-indexkeyed by column name.Depth matters more here than in a thin-leaf tree — a stratum leaf is fat (each
ChunkEntrycarries its wholePersistentColChunk), so:with-leavesis materializing the column data, while:interior(default) warms just the spine and makesChunkStats-driven pruning free.Tests are oracle-based on
CachedStorage's miss counter: full warm → the following scan costs zero further reads; budget is a hard ceiling (exactly N) split evenly across columns;:interior≪:with-leaves; warming never makes a scan more expensive. One boundary was measured rather than assumed and is deliberately not asserted: stratum's blob granularity is coarser than its node structure, so which blob a scan touches is storage-layout, not warm contract — the test file documents this.Bumps PSS
0.5.141 → 0.5.142(both pins). Full suite on the new PSS: 1435 tests, 5884 assertions, 0 failures.Next in the arc: proximum gets a search-path-shaped warm, then datahike's
-sec-warm!wiring (coordinated with the in-flight secondary PR) delegates here.