Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions deps.edn
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
;; `idx-stats` no longer restores the subtree to rebuild it (replikativ/persistent-sorted-set#24).
org.replikativ/persistent-sorted-set {:mvn/version "0.5.141"}
;; konserve for durable storage
org.replikativ/konserve {:mvn/version "0.9.350"
org.replikativ/konserve {:mvn/version "0.9.375"
:exclusions [org.clojure/clojurescript
com.github.pkpkpk/cljs-node-io
com.github.pkpkpk/fress
Expand Down Expand Up @@ -65,7 +65,11 @@
javax.servlet/javax.servlet-api]}}

:aliases
{:dev
;; Sibling checkout, for co-development across the replikativ stack.
;; Not required to build — konserve 0.9.375 carries `store-id`.
{:local {:override-deps {org.replikativ/konserve {:local/root "../konserve"}}}

:dev
{:extra-paths ["dev" "test" "bench" "notebooks"]
:extra-deps {nrepl/nrepl {:mvn/version "1.3.0"}
cider/cider-nrepl {:mvn/version "0.50.2"}
Expand Down Expand Up @@ -211,7 +215,7 @@
;; Use published deps (for CI/release)
:release
{:override-deps {org.replikativ/persistent-sorted-set {:mvn/version "0.5.141"}
org.replikativ/konserve {:mvn/version "0.9.350"
org.replikativ/konserve {:mvn/version "0.9.375"
:exclusions [org.clojure/clojurescript
com.github.pkpkpk/cljs-node-io
com.github.pkpkpk/fress
Expand Down
116 changes: 116 additions & 0 deletions doc/gc-and-store-ownership.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
# GC and store ownership

Why stratum's garbage collection looks different from datahike's, scriptum's and
proximum's, and what would have to be decided to make them the same.

Written while landing `konserve.gc-guard` across the stack.

## Two patterns already exist

| adapter | where its data lives | `mark-from-key-map` |
| --- | --- | --- |
| stratum | datahike's store (shared) | real — contributes its reachable keys |
| proximum | its own store (`:store-config`) | `#{}` |
| scriptum | its own path (`<store>-ft` sidecar) | `#{}` |

## Consistency splits into two axes

**Axis A — the collection mechanism.** Guard the values-then-pointer window,
derive the sweep cutoff from `konserve.gc-guard/cutoff` *before* marking, sweep
with `konserve.gc/sweep!`.

- datahike: yes
- scriptum: yes (as of the konserve backing)
- proximum: yes (as of `feat/konserve-gc-guard`)
- stratum: no — a per-store JVM monitor, no timestamp cutoff, deletion limited
to three explicitly enumerated key families

**Axis B — store ownership.** Own your store and sweep it (scriptum, proximum)
versus share a store and contribute marks (stratum).

## Stratum's position on A follows from its position on B

`konserve.gc/sweep!` is allow-list: it deletes every key that is not in the
whitelist and is older than the cutoff. That is only safe for a store you own
outright.

Stratum shares datahike's store — stated in
`datahike/index/secondary/stratum.clj`:

> Stratum shares datahike's store […] stratum writes to datahike's konserve
> store, so datahike's GC must preserve stratum's keys.

So stratum's explicit-family deletion is not an oversight; it is the adaptation
that sharing forces. Porting it to `sweep!` would require a whitelist covering
every key datahike owns, in the one configuration where getting that wrong
destroys another library's data.

Stratum is already safe on both of its paths:

- **Embedded**: datahike opens `guard/writing!` *before* the secondary flush —
`datahike/writing.cljc` names stratum in the comment, because `-sec-flush`
writes konserve keys from inside it — and absorbs stratum's marks via
`sec/mark-from-key-map :stratum`.
- **Standalone**: `with-storage-lock` serializes `gc!` against `sync!`.

## The decision is B, not A

**Model 1 — every library owns its store.** Stratum moves out of datahike's
store, adopts A wholesale, and its `mark-from-key-map` becomes `#{}` like the
others.

- Buys: one mechanism everywhere; no cross-library mark protocol; a mark bug can
only damage its own data.
- Costs: N stores to configure, back up and replicate; no structural sharing
across libraries; a migration for stratum.

**Model 2 — one store, many marks.** Scriptum and proximum move into datahike's
store and implement real marks.

- Buys: one store to operate; one GC; secondary data co-located with the datoms
that reference it.
- Costs: every `mark-from-key-map` must be exhaustively correct or datahike's
sweep deletes that library's data. One library's mark bug becomes another's
data loss.

Two things that landed this week change what is available here. Scriptum's
konserve backing makes Model 2 *possible* for it at all — under the path design
its data was files, which is why its `-sec-mark` still reads "Scriptum uses
filesystem, not konserve — nothing to mark". And `konserve.gc-guard` makes
Model 2 *safe*, being the shared safe point across libraries in one store.

Weak recommendation: **Model 1.** Failure containment is worth more than
operational tidiness, and landing this work surfaced two independent mark bugs —
proximum's mark never whitelisted branch heads, so every GC deleted the head
commit of every branch. Model 2 asks you to bet against exactly that class of
bug.

## Tripwire

If scriptum's konserve backing ever shares datahike's store,
`mark-from-key-map :scriptum` returning `#{}` becomes a data-loss bug. It is
correct today only because scriptum owns its store.

## Fixed here: the GC/sync lock key

`with-storage-lock` allocated its monitor in a map keyed by the store
*reference*, reasoning that record equality would make two connections to one
path share a lock. Measured, they do not:

```
identical? false = false same hash? false
```

So a process that connected twice got two independent monitors and no exclusion
at all — 0.16 ms of contention where the lock should have blocked for 300 ms.

The accompanying claim that "under-locking is impossible: distinct keys always
get distinct locks" was backwards. Two distinct keys for one physical store *is*
under-locking. The safe direction is the opposite, and it is the same rule the
GC safe point follows: the key may be **coarser** than the physical store — two
stores collapsing onto one monitor merely over-locks — but never **finer**.

The lock now keys on konserve's `:id`, which is coarser (a logical identity,
shared by replicas across machines) and within one JVM names one store. Stores
built through a backend constructor carry no id and fall back to the object,
which is the old behaviour.
Loading
Loading