shale

Horizontal-scale layer for embedded KV stores. Each instance you import is a cluster node; consistent hashing distributes keys across nodes; durability is whatever backend you plug in.

Why it exists

Sharded KV stores either:

• Run as heavyweight services (Cassandra, ScyllaDB, TiKV) - operational overhead, separate process
• Bake clustering into the data engine (Redis Cluster) - locks you to that engine
• Don't exist in the embedded-library + pluggable-backend shape

shale is the third option: thin Go library, BYO backend, scales by adding processes that import it.

Roadmap

• v0.1: single-node Cluster, gRPC service, shaled standalone binary, shale CLI (put/get/delete/scan/topology/stats/ping). API lockup.
• v0.2: multi-node hash ring + gRPC forwarding (static topology); hash-tag co-location; topology RPC + CLI subcommand. (v0.2.1, v0.2.2 followed as bug-fix rolls.)
• Pebble backend (pure Go, durable across restart); slots into the Backend interface alongside memory + slate.
• shale bench subcommand (originally roadmapped for v0.5; pulled forward to help measure v0.3 rebalancing impact).
• v0.3: shard rebalancing on membership changes (settle-delay, streaming MigrateRange RPC, write rejection during cutover, in-flight read forwarding, cleanup sweep); shale rebalance CLI. (v0.3.1 followed with 7 fixes including a P0 data-loss path on real backends.)
• v0.4: replication factor R + tunable read/write consistency (Nearest / Quorum / All) + LWW conflict resolution + read-repair + Delete-as-tombstone. (v0.4.1 fixed 8 review concerns; v0.4.2 eliminated a membership data race; v0.4.3 stabilizing test fixtures.)
• v0.5: observability (Prometheus metrics, tracing hooks) + real benchmarks vs single-node backends.
• v0.5.x: multi-module monorepo split (backends/slate + backends/pebble each get their own go.mod); backend-specific Settings pass-through for engine knobs (e.g. slate's await_durable); thin-shell cmd/shaled with per-backend builds. Lands before v0.6 freezes the API.
• v0.6: hostthis migration: swap raw SlateDB for shale-with-SlateDB-backend on production-shape data. Default to WriteOptions{AwaitDurable: false} paired with ReplicationFactor=3 (the relaxed durability mode validated by the v0.5 bench: ~150x put throughput vs strict at the cluster-n3-r3 production shape, with a ~100ms per-replica loss window made tolerable by 3-way replication).

Don't use in production yet.

Development

One-time setup after cloning, to enable the repo-shipped lint + format hooks:

git config core.hooksPath .githooks

This points git at .githooks/ for hook lookup, so the pre-commit script in that directory runs on every commit. It checks gofmt, runs go vet, and (when golangci-lint is installed) lints the staged diff. See .githooks/README.md for the install command and version pin.

License

Apache 2.0. See LICENSE.