Optional impact-ordered lexeme rank index for single-word FTS ranking

[samuelvkwong]

What

An opt-in fast path for the FTS arm: pgsearch_lexemerank stores, for every report and every lexeme of its tsvector, the exact ts_rank value a single-term query of that lexeme would score — an impact-ordered posting list. With the flag on, a single-word search reads its top HYBRID_FTS_MAX_RESULTS rows straight off the (lexeme, rank DESC, report) btree instead of computing ts_rank over every matching row and sorting. Every other query shape (multiple words, phrases, boolean operators) keeps the existing ts_rank path — deliberately, see below.

Ranking is exact-parity, not approximate: a row exists iff the lexeme is in the document's tsvector (membership ≡ search_vector @@ lexeme), and the stored value is the very number the fallback would compute, so the ordering is identical and RRF fusion downstream is unchanged.

Measured on the staging corpus (1.7 M reports), against the per-arm timings of #286:

query	ts_rank today	via lexeme rank index	speedup
fraktur (414,572 matches)	12,087 ms	27–42 ms	~400×
fraktur & befund (befund: 1.6 M matches)	6,440 ms	4,643–4,977 ms¹	~1.3×

¹ spike measurement of the posting-list join; this PR does not even attempt it (see limitations).

Why this works for single-term queries

ts_rank's cost problem is when it runs: it can only score a document given the full tsvector, so the ranked query detoasts and ranks every match before ORDER BY … LIMIT can cut the list (11–13 s for common German terms; the GIN index can confirm membership cheaply but cannot return rows in rank order).

For a single-term query, though, the score is a pure function of (document, lexeme) — nothing else in the query influences it. That makes it precomputable at write time: the sync trigger unnests the tsvector and stores ts_rank(search_vector, 'lexeme') per lexeme once per report write (amortized ~µs–ms per report; the spike built all 414 k fraktur rows in 5.6 s). At query time "top N documents for this word" is then a bounded walk of the (lexeme, rank DESC) btree — the index is the sort.

Why this cannot work for multi-term queries

A multi-term score is a per-query aggregate: for fraktur & befund, a document's rank is the combination of both contributions, and it only exists for documents satisfying the boolean expression. Precomputing it for every possible term combination is combinatorially impossible, so it must be assembled at query time:

• The relational plan is intersect-then-sort. Join the 414 k-row fraktur posting list with the 1.6 M-row befund list, combine ranks, sort, take the top N. Measured: ~4.8 s — barely better than ts_rank's 6.4 s, because the cost was never the per-document arithmetic; it is touching every candidate before the limit.
• The known fix is procedural, and SQL cannot express it. Engines that do this fast (Lucene, Tantivy) walk all posting lists impact-ordered simultaneously, document-at-a-time, maintaining the current Nth-best score as a threshold and using per-term/per-block score ceilings to prove "no document still ahead can beat the threshold" — then stop (WAND / Block-Max WAND). That early termination is a stateful traversal with data-dependent stopping; there is no SQL formulation of "stop scanning when f.rank + max(remaining b.rank) < threshold", and PostgreSQL's executor has no operator that prunes one scan using another scan's running top-N.
• Phrases and negations don't even have the inputs. The table stores no positions, so "a <-> b" cannot be verified from it; NOT/OR change membership per query. All of these shapes short-circuit to the ts_rank path before touching the table.

This boundary is exactly where hand-rolling ends and #285 (pg_textsearch) begins: the extension keeps impact-ordered postings and implements Block-Max WAND in C, generalizing this PR's 27 ms to arbitrary boolean/multi-term queries, with real corpus IDF on top. The numbers above double as evidence for it — they demonstrate the achievable ceiling on this corpus and why SQL alone cannot reach it beyond one term.

Design notes

• Opt-in: HYBRID_FTS_LEXEME_RANK_INDEX=false by default; nothing changes for existing deployments (table is created empty, no trigger, no write cost).
• Enable = set the flag + run ./manage.py sync_lexeme_ranks once (installs the sync trigger, backfills in batches, resumable; --rebuild, --remove provided).
• Safety: system check pgsearch.E006 fails migrate when the flag is on but nothing maintains the table; at runtime the provider probes for the trigger once per process and falls back to ts_rank with a WARNING rather than serve from a stale table.
• Costs: storage (est. 15–25 GB at staging scale for the full corpus) and write amplification on report (re)indexing (the trigger recomputes that report's rows). A follow-up could restrict rows to high-document-frequency lexemes — rare terms are already fast through ts_rank because they have few matches.
• Embedding-only writes (UPDATE … SET embedding) do not fire the trigger.

Tests

10 new tests in test_lexeme_rank.py: exact order/count parity with the ts_rank path (single-language and all-configs), stored value == ts_rank value, dispatch (multi-term/phrase/stopword/flag-off never touch the table), trigger freshness on create/update, missing-trigger fallback + warning, --remove. Full pgsearch + search suites: 248 passed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VDei6anDxfR5eoHhFfXBGs

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[samuelvkwong]

Live results from the staging deployment (1.7 M-report corpus, full backfill: 139,885,956 rank rows / 28 GB, ~10.5 h single worker; trigger active throughout):

query (matches)	fts_ms before, ts_rank	fts_ms after, lexeme index	speedup
Fraktur (414k)	12,800–14,200	248–308	~45×
Befund (1.6 M)	~15,000 (est. from match ratio)	522	~30×
Krebs (26)	46–90	43	parity
Fraktur Befund (multi-word)	12,200	12,206 — correct fallback	by design

The plan is the intended one: index-only scan of pgsearch_lexeme_rank_idx in (rank DESC, report_id) order (10,000 entries in ~21 ms), streaming nested-loop probes for group access, Unique over the sorted stream, no sort node. The gap between the ~30 ms index walk and the ~250–500 ms end-to-end fts_ms is the per-emitted-row access-control joins (group M2M + report + language probes × 10,000) — a structural cost of the window size, not of the index.

One operational note: run VACUUM (ANALYZE) (or let autovacuum catch up) after a large backfill — until the visibility map is set, the index-only scan does heap fetches.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VDei6anDxfR5eoHhFfXBGs