Skip to content

Dataset warming via persistent-sorted-set 0.5.142 - #41

Merged
whilo merged 5 commits into
mainfrom
feat/warm
Aug 24, 2026
Merged

whilo merged 5 commits into
mainfrom
feat/warm

Conversation

@whilo

@whilo whilo commented Aug 23, 2026

Copy link
Copy Markdown
Member

First consumer of persistent-sorted-set#25's warm push-down, and the reason the multi-tree budget round-robin exists: stratum.warm/warm! warms every indexed column of a dataset in one call, all columns sharing one budget so none starves another; warm-column! scopes to one. Report is PSS's, :by-index keyed by column name.

Depth matters more here than in a thin-leaf tree — a stratum leaf is fat (each ChunkEntry carries its whole PersistentColChunk), so :with-leaves is materializing the column data, while :interior (default) warms just the spine and makes ChunkStats-driven pruning free.

Tests are oracle-based on CachedStorage's miss counter: full warm → the following scan costs zero further reads; budget is a hard ceiling (exactly N) split evenly across columns; :interior ≪ :with-leaves; warming never makes a scan more expensive. One boundary was measured rather than assumed and is deliberately not asserted: stratum's blob granularity is coarser than its node structure, so which blob a scan touches is storage-layout, not warm contract — the test file documents this.

Bumps PSS 0.5.141 → 0.5.142 (both pins). Full suite on the new PSS: 1435 tests, 5884 assertions, 0 failures.

Next in the arc: proximum gets a search-path-shaped warm, then datahike's -sec-warm! wiring (coordinated with the in-flight secondary PR) delegates here.

whilo and others added 5 commits August 16, 2026 23:24
`with-storage-lock` is what keeps stratum's standalone `gc!` from
sweeping chunks a concurrent `sync!` is about to reference. It allocated
its monitor in a ConcurrentHashMap keyed by the store REFERENCE, on the
stated reasoning that

  "File-store / S3-store records with content-based equality: two
   instances configured against the same backing storage share a lock —
   which is what we want (they protect the same physical data)."

They do not. Two `connect-fs-store` calls on one path are neither
`identical?` nor `=`, and hash differently:

  identical? false    = false    same hash? false

So a process that connected twice to one store got two independent
monitors and no exclusion at all — measured at 0.16ms of contention
where the lock should have blocked for 300ms.

The companion claim, that "under-locking is impossible: distinct keys
always get distinct locks", had it exactly backwards. Two distinct keys
for ONE physical store IS under-locking. The safe direction is the
opposite, and it is the same rule `konserve.gc-guard` follows for the
safe point: the key may be COARSER than the physical store — two stores
collapsing onto one monitor merely over-locks — but never FINER.

konserve's `:id` is coarser: a logical identity shared by replicas
across machines, and within one JVM (all a monitor can reach) one id
names one store. Stores built through a backend constructor carry no id
and fall back to the object, which is the old behaviour; such callers
should still avoid connecting twice.

The regression test observes the monitor through the behaviour it exists
for rather than reaching into private state: a lock held on one
connection must exclude the other. It fails on the old key.

Signed-off-by: Christian Weilbach <christian@weilbach.name>
…unify it

Companion to the lock fix. Records that stratum's GC design follows from
sharing datahike's store rather than owning its own, that both patterns
already exist in the stack, and that the open question is store ownership
rather than collection mechanism.

Signed-off-by: Christian Weilbach <christian@weilbach.name>
Carries `konserve.protocols/store-id`, which the GC/sync lock now keys on
(replikativ/konserve#159). Both pins bumped — the main dependency and the
`:release` override. The `:local` alias stays for co-development but is no
longer needed to build.

1430 tests / 5866 assertions against the published artifact.

Signed-off-by: Christian Weilbach <christian@weilbach.name>
`stratum.warm/warm!` warms every indexed column of a dataset in one call:
each column contributes its PSS tree and all of them share ONE budget
round-robin (persistent-sorted-set's `warm-trees!`, new in 0.5.142), so no
column can spend the budget before another gets any — which column the next
query needs is not knowable at warm time. `warm-column!` scopes to one.

Depth is a bigger lever here than in a thin-leaf tree, because a stratum leaf
is FAT: each ChunkEntry carries its whole PersistentColChunk, so restoring the
leaves IS materializing the column data. `:depth :interior` (the default)
warms the spine — branches carry the ChunkStats measures, so stats and
measure-driven pruning answer without touching a chunk — and `:with-leaves`
with a sized `:budget` prefetches the data a known scan will read.

The tests' oracle is CachedStorage's own miss counter, with one boundary
learned by measuring rather than assuming: stratum's blob granularity is
COARSER than its node structure (a whole small tree can be one or two blobs),
so which blob a scan touches is a storage-layout fact and deliberately not
asserted. What is asserted is the warm's contract — a full warm makes the scan
that follows cost ZERO further reads, the budget is a hard ceiling (exactly N,
not one more) split evenly across columns, :interior stays a small fraction of
:with-leaves, and warming never makes a scan more expensive.

Full suite on 0.5.142: 1435 tests, 5884 assertions, 0 failures.

Signed-off-by: Christian Weilbach <ch_weil@topiq.es>
doc/warming.md is the user-facing story: the problem (a lazy-loaded dataset
pays misses x RTT with nothing overlapping), the two entry points, why depth
is a bigger lever here than in a thin-leaf tree (a stratum leaf carries its
chunks, so :with-leaves IS materializing the column), the report as an early
metric (:budget-exhausted? shows a decaying warm before p99 does), and —
stated as prominently as what warming does — what it does not promise: results
never change, :fetched counts restores not misses, and blob granularity is
stratum's rather than the tree's. Linked from the README with a short section
in the reading path between the quick start and durability.

doc/gc-and-store-ownership.md was a development artefact — "written while
landing gc-guard", present-tense drift like "landed this week" — describing
its cross-project relationships more than any decision a reader could act on.
The content deserves better: it records WHY stratum's GC differs structurally
(shared store => contribute marks, owned store => sweep) and an ecosystem
decision that is still open. Reframed as reference documentation and linked
from the README's Ecosystem section, which is where a reader wondering about
the shared store actually is.

Signed-off-by: Christian Weilbach <ch_weil@topiq.es>
@whilo
whilo merged commit bac4366 into main Aug 24, 2026
5 of 6 checks passed
@whilo
whilo deleted the feat/warm branch August 24, 2026 00:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant