graph projection retains the whole source table: budget-exempting it trades a stale graph for a daemon OOM; needs finite budget now, incremental+chunked projection as the durable fix

[bgmcmullen]

Prod fallout is tracked in hyparam/hypaware-server#143: the fleet graph has been frozen at 2026-07-10 because every automatic projection sweep dies on the query heap budget. The root cause and the durable fix both live in this repo.

Root cause

projectGraph (hypaware-core/plugins-workspace/context-graph/src/project.js) runs three query shapes through executeQuerySql without a maxHeapBytes override, so they inherit the default 1GiB user-query heap budget added in #295:

• the shared per-contract scan (project.js:99): SELECT <union of rule columns> FROM ai_gateway_messages, no WHERE, fully materialized via collect(results)
• the raw-SQL rule scans (project.js:129)
• the dedup id read (project.js:204)

The budget watchdog force-GCs before refusing, so a trip means the projection genuinely retains over 1GiB of live heap. That is by construction: the shared scan materializes the entire source table into result.rows before the rule loop sees a single row. On prod (~1.66M rows, ~50k/day growth) that crosses the budget, the server scheduler logs graph_projection.scope_failed, re-marks the scope dirty, and fails identically on every subsequent sweep. The deploy that enabled automatic projection (server #124) shipped alongside the guard that kills it (#295).

Why the obvious remedies are wrong

• maxHeapBytes: 0 at the three call sites (proposed on the server issue): removes the guard without removing the O(table) retention. At current growth the sweep OOMs the daemon in months, and an OOM at sweep time recurs every sweep: a daily crash-loop that takes down query serving for the whole org. Strictly worse than a stale graph. The budget did not create this risk; it surfaced a pre-existing one.
• Raising HYP_QUERY_MAX_HEAP_MB on the daemon: global, so it strips protection from real POST /v1/query user traffic, the crasher class the budget exists for.

Plan

1. Immediate unblock: dedicated finite budget (small PR)

Pass an explicit maxHeapBytes at the three projection call sites, sourced from its own knob (e.g. HYP_GRAPH_PROJECTION_MAX_HEAP_MB, default ~3GiB), never 0. Unfreezes prod today, leaves the 1GiB user-query guard untouched, and keeps the fail-clean property: when the table outgrows it we get graph_projection.scope_failed in the log again instead of a daemon crash-loop. Acknowledged treadmill; it exists only to buy time for step 2.

The server-side regression test staged on hypaware-server#143 (test/graph-projection-budget.js) should be adjusted to expect a finite dedicated budget rather than an exemption, and goes green once this ships.

2. Durable fix: incremental projection + chunked rebuild (LLP design change)

Two distinct problems, two mechanisms:

• Bounded memory (chunking). Peak memory must scale with a chunk plus the node/edge accumulator maps (entity-count-sized, ~10k today), not with message volume. mergeRow is documented order-independent precisely so row feed order cannot matter, so chunked processing is semantically safe by design. Chunk boundaries must follow the cache's physical partitions (discoverCachePartitions in src/core/cache/), not SQL WHERE predicates: N predicate chunks that do not prune files re-scan all files N times, reintroducing exactly the read amplification LLP 0095 removed.
• Bounded daily work (incrementality). Even a chunked full rescan reads the whole table every sweep to re-derive rows that dedupExisting then drops: any row whose content-addressed id is already committed contributes nothing (first write of an id wins until compaction). A sweep that scans only source rows past a watermark commits essentially identical output. The cache already provides the primitive: the cache-global strictly-increasing ingest sequence (src/core/cache/ingest-seq.js) guarantees each partition observes an increasing subsequence of seqs, the same property the sink watermark relies on. Daily sweep scans rows past the last projected seq (small, fast, trivially within budget); the chunked full rescan remains as the rebuild path for bootstrap, rule changes, or corruption recovery.

Watermark care: seq-based, not timestamp-based (late-arriving rows), and a failed sweep must not advance the seq.

This step needs an LLP doc (touches LLP 0095/0096/0097 territory) landed with the code, plus @ref updates at the shared-scan site.

Acceptance

• Prod graph advances daily again with the default budget knob (step 1) and stays within budget regardless of table size (step 2).
• User-query budget semantics unchanged.
• hypaware-server#143's regression test green against the updated client.

🤖 Generated with Claude Code