fix(literature): sort candidates by keyword overlap before char-cap truncation

[akkiittiwari]

Summary

• Stage 5 (LITERATURE_SCREEN) was rejecting all candidates on niche/cross-domain topics. Root cause: candidates.jsonl retains raw search order (whichever query/source happened to return a paper), and _MAX_CANDIDATES_CHARS truncates the serialized candidate list before it's sent to the screening LLM. On a real run with 1418 filtered candidates, the 30,000-char cap only covered the first ~15-20 rows — all off-topic noise pulled in by a low-signal expanded query (e.g. "The Stated vs. Revealed survey"). Every relevant paper sat past index 246 and never reached the screening prompt, so the LLM correctly (but uselessly) rejected the unrepresentative slice it was shown, and the stage paused with model_rejected_all.
• Fix: sort filtered_rows by keyword_overlap (already computed during the pre-filter) descending before truncating, so relevant candidates survive the cap regardless of raw search order. Also raised _MAX_CANDIDATES_CHARS from 30k to 100k chars so more of the relevant pool gets through.

Verification

• Replayed the fix against the actual candidates.jsonl (1495 papers) from a run that had been stuck for 3 HITL-approval cycles in a row with an empty shortlist. Post-fix, the top of the truncated candidate list is dominated by genuinely on-topic papers (45 relevant candidates fit under the new cap, well above the pipeline's 15-paper minimum shortlist size).
• Re-ran the stuck pipeline end-to-end from the Stage 5 checkpoint: it produced a real 26-paper shortlist (top hit: 0.9 relevance) and progressed through Stages 6-9 (KNOWLEDGE_EXTRACT, SYNTHESIS, HYPOTHESIS_GEN, EXPERIMENT_DESIGN) for the first time, versus previously looping indefinitely on Stage 5.
• pytest tests/test_rc_literature.py (run under the project's .venv, Python 3.11) — 47 passed.

Test plan

• pytest tests/test_rc_literature.py (project .venv)
• Manual replay of Stage 5 filtering/sorting logic against a real stuck-run candidate set
• Live pipeline resume from checkpoint through Stage 9