Disk KV cache: cold/evict checkpoints get no eviction-score advantage over continued

[unsaltedbutter-ai]

TL;DR

kv_entry_eviction_score is (hits+1) * tokens / size regardless of reason. A freshly written cold or evict checkpoint loses the eviction race to a competing workload's fresh continued snapshots on density alone. In production a 1.3 GiB live-state save died within ~5 min while a second conversation prefilled.

How it fails today

When conversation B writes a continued, A will have a cold, an evict, & a continued on disk.
The score function gives all three the same weight, so the victim depends on incidental density differences and not on intent. If cold & evict were boosted, the continued would be the consistent target.
Then if conversation A resumes right where it left off the evict would cache hit and a significant amount of prefill time would be avoided.

In a production case the live-state evict was 1.3 GiB and got unlinked after 5m 19s while B prefilled.

Proposed fix

Bump the base from 1.0 to 4.0 for KV_REASON_COLD | EVICT | SHUTDOWN in kv_entry_eviction_score. Continued and unknown stay at 1.0. A continued must serve three real hits before outranking a fresh same-shape checkpoint. Existing hit decay still applies.

[antirez]

Another angle to look at this is to recompute the score in a way that makes large files without recent hits scoring much lower. I'll look at this with more care, I'm in the middle of some work I can't stop. Thanks.

[unsaltedbutter-ai]

The idea is not that large files without hits aren't as valuable, it is that continued is less valuable than evict and cold regardless of size.

Cold would be if we started a new conversation from that same client again. (Cold hits)
Evict would be if we resumed an existing conversation that was interrupted by some other conversation.

On main, before #174 (#175) and #176 (#177) the kv cache looks like:

• conversation A cold
• conv A contine 10k
• conv A contine 20k
• conv A contine 30k
• ...

Then when conversation b comes in the kv cache adds:

• conv A evict ##k

Then conv b starts writing continues and the A-evict and A-cold has an equal weighted chance of being unlinked as A-continued.

If we were to resume conv-A after conv-B and the A-evict had been unlinked the cache has failed us.

With #174/#175, when conv-B starts writing continues, the cache would look like because A-continue n-1 gets unlinked at the time A-continue n gets written:

• conversation A cold
• conv A contine 30k
• conv A evict ##k

With #176/#177 the conv-A continue-biggest would be unlinked first, preserving the conv-A evict at least until conv-B starts writing continues > the size of conv-A's biggest continue.