Skip to content

perf(translate,turso): borrowed-key Arc cache + opt-in Turso handle reuse - #35

Merged
luthermonson merged 1 commit into
mainfrom
perf/translate-cache-and-turso-reuse
Aug 16, 2026
Merged

perf(translate,turso): borrowed-key Arc cache + opt-in Turso handle reuse#35
luthermonson merged 1 commit into
mainfrom
perf/translate-cache-and-turso-reuse

Conversation

@luthermonson

Copy link
Copy Markdown
Contributor

What

Two independent perf changes to the translate cache and the experimental Turso backend.

1. TranslateCache: borrowed-key + shared Arc

Each dialect gets its own LRU keyed by a borrowed &str (Box<str> keys, Box<str>: Borrow<str>), returning a shared Arc<[TranslateResult]>. This removes, on every cache hit:

  • the (Dialect, String) key allocation that a tuple key forced on each lookup, and
  • the deep Vec<TranslateResult> clone handed to the caller.

Callers (litewire-session, litewire-postgres) now iterate the shared slice by reference; translate_cached returns the Arc.

Sharding was measured and dropped. An LRU get is a write (it promotes to MRU), so concurrent hits on a hot statement contend on that entry's list links regardless of how many locks guard the map — aggregate throughput did not improve — while the extra hash to select a shard cost more than the lookup, and dividing a fixed capacity across shards turned a skewed key distribution into thrashing.

translate-cache hit, single-thread hot key (release, median 9 × 2M) ns/op
before — owned String key + Vec clone 69.8
after — borrowed &str key + Arc clone 23.8

2.93×.

2. litewire-turso: opt-in idle-connection reuse

TursoBuilder::handle_reuse(max_idle) (off by default). A clean connection is parked on drop and checked back out by a later session, keeping the engine's per-connection prepared-statement cache warm. A fresh Database::connect() opens with an empty statement cache, and the first prepare of each distinct statement on a cold connection is the dominant per-session cost — the whole prize is amortising it.

Hygiene is cheap per-statement dirty-tracking that gates parking, not an expensive checkout probe:

  • A wire client's SET ... translates to a no-op, so the only way to change connection-scoped pragma state is a literal PRAGMA name = value — caught generically.
  • CREATE TEMP/TEMPORARY (a sqlite_temp_master object) and ATTACH/DETACH are the remaining vectors.
  • Open transactions are refused at park (is_autocommit, sync); last_insert_rowid is snapshot at checkout so a reused handle never reports the prior session's insert id (the engine has no setter).

Checkout is a free autocommit re-confirm — no queries.

An earlier design validated a PRAGMA-readback fingerprint at checkout (temp-count + N pragma reads). Measured 0.99× — no net win: the probe cost as much as the statement-cache saving. That result drove the redesign above.

turso session cycle: connect + 5 distinct first-queries + drop (file-backed, single-thread, median) µs/cycle
reuse OFF (fresh connect, cold cache) 154.9
reuse ON (parked warm handle) 53.0

2.92×.

MySQL wire session: TCP + handshake + 5 SELECTs + close (end-to-end via mysql_async, median) µs/session
reuse OFF 1460
reuse ON 1220

1.20× — reuse saves ~240 µs/session; the absolute is dominated by the per-session TCP + MySQL handshake, which reuse does not touch.

Adds ReuseStats + Turso::reuse_stats(). Reuse is ignored (with a warn!) when enable_cdc_on_connect is set.

Tests

Reset-behaviour tests for every leak vector: temp objects, pragma writes, open transactions, last_insert_rowid, plus staleness age-out, pool-capacity bound, and reuse-disabled-by-default. cargo test --workspace --all-features green (956); clippy -D warnings clean on default and --all-features; nightly fmt clean.

Notes

  • No dependency or Cargo.toml changes. Turso backend remains experimental.
  • Measurements are single-threaded medians (the reproducible lane; contended throughput swings run to run).

…euse

TranslateCache: key each dialect's LRU by a borrowed &str and hand back a
shared Arc<[TranslateResult]> instead of allocating a (Dialect, String) key
per lookup and deep-cloning a Vec on every hit. One lock per dialect; no
sharding — sharding measured a net loss: an LRU get is a write (it promotes
to MRU), so concurrent hot-key hits contend on that entry's list links no
matter how many locks guard the map, while the second hash needed to pick a
shard costs more than the lookup it parallelises.

  hot single-key hit, release, median of 9 x 2M iters:
    before (owned String key + Vec clone)   69.8 ns/op
    after  (borrowed &str key + Arc clone)   23.8 ns/op   2.93x

litewire-turso: opt-in idle-connection reuse (TursoBuilder::handle_reuse).
A clean connection is parked on drop and checked back out by a later session,
keeping the engine's per-connection prepared-statement cache warm — a fresh
connect() opens with an empty cache and the first prepare of each statement
is the dominant per-session cost. Hygiene is cheap per-statement
dirty-tracking (literal PRAGMA writes, CREATE TEMP, ATTACH/DETACH — a wire
client's SET translates to a no-op, so those are the only vectors) that gates
*parking*; checkout stays free. last_insert_rowid is snapshot at checkout so
a reused handle never reports the prior session's insert id (the engine
exposes no setter).

An earlier design that validated a PRAGMA-readback fingerprint at checkout
measured 0.99x — the probe cost as much as the statement-cache win it
protected — which drove the move to park-time dirty-tracking:

  file-backed session cycle (connect + 5 first-queries + drop), median:
    reuse OFF  154.9 us    reuse ON  53.0 us    2.92x
  MySQL wire session (TCP + handshake + 5 SELECTs + close), end-to-end:
    reuse OFF  1460 us     reuse ON  1220 us    1.20x

Reuse is disabled by default and ignored when enable_cdc_on_connect is set.
Adds ReuseStats + Turso::reuse_stats() for observability, and reset-behaviour
tests for the temp-object, pragma-write, open-transaction and
last_insert_rowid leak vectors.
@luthermonson
luthermonson merged commit defab22 into main Aug 16, 2026
5 checks passed
@luthermonson
luthermonson deleted the perf/translate-cache-and-turso-reuse branch August 16, 2026 17:58
luthermonson added a commit that referenced this pull request Sep 1, 2026
…euse (#35)

TranslateCache: key each dialect's LRU by a borrowed &str and hand back a
shared Arc<[TranslateResult]> instead of allocating a (Dialect, String) key
per lookup and deep-cloning a Vec on every hit. One lock per dialect; no
sharding — sharding measured a net loss: an LRU get is a write (it promotes
to MRU), so concurrent hot-key hits contend on that entry's list links no
matter how many locks guard the map, while the second hash needed to pick a
shard costs more than the lookup it parallelises.

  hot single-key hit, release, median of 9 x 2M iters:
    before (owned String key + Vec clone)   69.8 ns/op
    after  (borrowed &str key + Arc clone)   23.8 ns/op   2.93x

litewire-turso: opt-in idle-connection reuse (TursoBuilder::handle_reuse).
A clean connection is parked on drop and checked back out by a later session,
keeping the engine's per-connection prepared-statement cache warm — a fresh
connect() opens with an empty cache and the first prepare of each statement
is the dominant per-session cost. Hygiene is cheap per-statement
dirty-tracking (literal PRAGMA writes, CREATE TEMP, ATTACH/DETACH — a wire
client's SET translates to a no-op, so those are the only vectors) that gates
*parking*; checkout stays free. last_insert_rowid is snapshot at checkout so
a reused handle never reports the prior session's insert id (the engine
exposes no setter).

An earlier design that validated a PRAGMA-readback fingerprint at checkout
measured 0.99x — the probe cost as much as the statement-cache win it
protected — which drove the move to park-time dirty-tracking:

  file-backed session cycle (connect + 5 first-queries + drop), median:
    reuse OFF  154.9 us    reuse ON  53.0 us    2.92x
  MySQL wire session (TCP + handshake + 5 SELECTs + close), end-to-end:
    reuse OFF  1460 us     reuse ON  1220 us    1.20x

Reuse is disabled by default and ignored when enable_cdc_on_connect is set.
Adds ReuseStats + Turso::reuse_stats() for observability, and reset-behaviour
tests for the temp-object, pragma-write, open-transaction and
last_insert_rowid leak vectors.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant