[七问法] Wiki-Link Retrieval：研究故事线与下一阶段验证合同

[ShuhaoZhangTony]

七问法研究合同

这是基于当前仓库证据的下一阶段建议。历史负结果继续保留；本合同不会把 smoke、replay、simulation、projected profile 或 host fixture 升格为端到端效果。

1. 问题定义：在大规模文档链接图上，能否只物化对当前查询有价值的局部子图，避免全量图物化的数量级开销？
2. 重要性：先在真实 workload/runtime 上量化该问题的发生频率、资源损失和 SLO/正确性影响；若 M0 不能复现可重复病理，则停止机制实现。
3. 现有工作缺口：已完成的全量 materialization 保持了正确性但 p95 开销约为 baseline 的 23.7 倍，说明全图预处理是错误抽象。
4. 核心机制假设：关闭全量配置，转向 query-conditioned lazy materialization 或增量局部更新；价值判定必须只使用查询时可见字段。
5. 最小可行设计：只实现能够区分 baseline/treatment 的最小真实 seam；所有 runtime、算法和数据依赖必须由本仓库 submodule 固定，禁止依赖 ambient shared core。先做 correctness gate，再申请扩大设备/模型矩阵。
6. 实验合同：以无图检索、全量物化旧机制和惰性局部物化为三臂；在 HotpotQA 等真实链接图上要求 evidence/answer oracle 不退化、访问边数下降至少 50%、端到端 p95 不高于 1.25 倍；否则停止新机制。 两臂必须使用同一 commit、模型、数据顺序、硬件和运行模式；至少 10 次重复或完整真实事件集，报告完整分布。正确性必须 100%，性能门须预注册；未过门只关闭本机制，不修改阈值迎合结果。
7. 预期知识增量：Link-graph retrieval is viable only when graph construction is query-scoped rather than globally materialized.

下一次提交要求

• 先在本 Issue 回复：架构/数据事实、matched baseline/treatment、指标阈值、correctness oracle、停止条件、证据等级、硬件与预计时长。
• 随后用独立分支提交最小实现、测试、原始结果、manifest 和 Draft PR。
• 不得把准备度检查、dry-run 或 projected 数字写成论文效果。

[ShuhaoZhangTony]

补充口径：负结果不等于课题终止

当前 no-go/负结果只关闭已经验证失败的机制、配置与故事线，不要求继续死磕旧方案，也不关闭这个研究领域。可以更换应用场景、因果变量、系统层次或观察视角，只要能找到真实且重要的问题。

建议下一步先做“场景发现”而不是参数调优：从真实 workload、trace、故障或架构特性中找可重复反例，再重新提交七问法合同。新方向需要说明它与旧机制的本质区别、为何现有工作仍解决不了、如何被实验否证。找不到真实场景时可以暂时停下；找到合理场景后，这仍然可以是很好的课题。

[ShuhaoZhangTony]

新假设：多 worker 的 query-scoped authored-link snapshot

1. 问题：link-aware retrieval 的 worker 是否必须在每次启动时把 46,873-node/142,706-edge authored graph 全量反序列化到私有内存？
2. 重要性：旧全图路径正确但相对 flat path 的 p95 为 23.662×；若开销主要来自每 worker 物化，扩大模型/worker 数会放大启动和内存成本。
3. 缺口：现有工作讨论候选扩展质量，较少把 authored graph 的 snapshot publication、共享读取和局部前沿访问作为独立系统问题。
4. 假设：原子发布、只读文件索引（先用 SQLite snapshot 验证）可让 worker 只读取活跃 source 的邻接表，避免 Python 全图 materialization，同时保持精确邻接语义。
5. matched arms：baseline=每个真实子进程解析同一 canonical HotpotQA JSONL 为完整 dict；treatment=同一 snapshot 的只读索引，每进程只读取固定活跃 source 集。输入、source 顺序、进程模型和 lookup 集一致。
6. 门槛/oracle：10 repeats；邻接集合与 digest 100% 一致，节点/边/metadata 守恒，无 partial snapshot；worker startup+lookup p95≤0.50、Python peak allocation≤0.50，单次 lookup p95≤1.25。报告 snapshot build 成本，不用未来 query label。
7. 证据边界/停止：真实 HotpotQA graph + host filesystem/process microbenchmark，不声明检索质量/serving/NPU。若两项资源门不过，只关闭该 snapshot 配置并换机制，不冻结 Wiki-Link。

预计 30–45 分钟，无加速卡和新依赖。

[ShuhaoZhangTony]

本轮真实结果已提交 PR #8：#8

canonical HotpotQA graph 46,873 nodes / 142,706 edges，AB/BA 各10轮，exact digest全通过。startup+lookup p95 ratio=0.0464，Python peak ratio=0.00617；但256-source lookup p95 ratio=59.9477，未过预注册≤1.25门，因此 all-cold snapshot 配置判定 no-go。下一假设改为 cold immutable snapshot + bounded adaptive hot frontier，并要求真实 query trace/保留的一阶段 rankings；不对本配置调参。

[ShuhaoZhangTony]

Pre-registered bounded follow-up (2026-08-16):

• Architecture/data fact: current main loads the authored-link graph once per worker and already performs query-local expansion. PR Measure query-scoped authored-link snapshot boundary #8 separately closed the all-cold SQLite snapshot configuration. This follow-up therefore tests only the documented remaining mechanism: an observed-query bounded hot tier over the same immutable snapshot.
• Trace: local public SGLang HotpotQA ReAct trace benchmark/react/hotpotqa_100.jsonl, source commit 70359bf31a4b86bb772cd0853cbf833faf159a7c, SHA256 41d0a75d73dd207a3fa3adb1440441d5a41442ed94eb19b2e38c3a00cc4519e9. Consume all 291 Search[...] actions from all 100 queries in original order. Ignore observations, Finish[...], rewards, answers, and supporting facts. Record query hashes rather than answer labels.
• Graph: current-main data/hotpotqa_authored_links.jsonl, SHA256 5d335ba19aa1f7aafaf13682a75e5c48d4c528ca3d243c8e94422cefaf26b355; preserve authored-link targets and order exactly.
• Arms: (E) current eager full Python adjacency loaded once per fresh worker; (S) immutable SQLite snapshot with every event served from the persistent substrate; (H) the same snapshot plus a fixed 64-source LRU hot tier populated only after an observed event. No future trace/label prewarming.
• Run: CPU/filesystem only, 10 matched repeats, cyclic counterbalanced E/S/H order, full raw event rows and atomic result. No installs, models, NPU, synthetic links, fuzzy title recovery, or parameter tuning.
• Oracle/gates: arm digests/frontiers/provenance/path order 100% equivalent; reopen 100%; errors, duplicate events, stale/cross-query state = 0. Pre-registered performance gates are H steady query p95 <= 0.50x E, H amortized end-to-end <= 1.25x E, and H steady query p95 <= 0.50x S. Report p50/p95/p99, complete distributions, Python peak RSS proxy, snapshot/hot-tier bytes, exact graph/trace hashes, and graph-title coverage.
• Stop/evidence: one formal run only; pass or fail closes only this 64-entry hot-tier configuration. Evidence is bounded real-public-query-trace-host-process-filesystem-runtime, not answer quality, serving, model, GPU, or NPU evidence.

[ShuhaoZhangTony]

Bounded configuration closed by merged PR #9 (90ebcc728099ddbdec1371add7122df7dfbba2e5). The sole run preserved frontier digest/found count/reopen across 10 matched repeats, but the fixed 64-entry tier observed only 5/291 hits per repeat. Hot event p95 was 0.042361 ms versus eager 0.002100 ms (20.1719x) and all-cold 0.035950 ms (1.17833x); hot/eager end-to-end p95 including startup was 0.059643. Both steady-query gates failed, so performance_claim=false; no capacity tuning, fuzzy matching, trace reorder, or rerun was performed. Evidence remains real-public-query-trace-host-process-filesystem-runtime, excluding answer/retrieval quality, serving, model, GPU, and NPU claims.

[ShuhaoZhangTony]

M0 code gate: canonical first-stage frontier trace (pre-implementation contract)

1. Causal problem: the failed 64-entry hot-tier run consumed natural-language SGLang Search[...] actions, not the canonical retriever’s actual document frontier; only 5/291 events hit the authored graph, so that trace cannot identify the locality seen by link expansion.
2. Why it matters: any claim about query-local graph storage depends on which authored-link source documents the real first stage exposes, in rank order. Inferring those sources from query strings confounds retrieval behavior with title matching.
3. Common assumption/gap: substrate evaluations often treat query text or answer-side labels as a proxy for graph keys. Current benchmark_multihop.py has the real pre-expansion ranking but does not expose a reusable, auditable event stream.
4. Engineering mechanism: add a default-off frontier-trace output at the actual canonical benchmark seam, immediately after first-stage ranking and before link expansion. Rows contain only query identity/hash, ordered visible seed document identities/ranks and graph-source membership; answers, supporting facts, qrels and future events are forbidden. Publication is atomic and fail-closed, with a versioned manifest binding input graph and run order.
5. Executable gate: actual-module tests must prove traced seeds equal the exact pre-expansion ranking/order, label fields are absent, disabled mode is byte-for-byte native, and any write/finalization error leaves no published partial artifact. This M0 does not run or tune a graph-storage treatment.
6. Falsifiable future evaluation: only after a complete canonical HotpotQA run emits this trace may a different substrate mechanism be preregistered against the same ordered frontier, with >=10 matched repeats, exact frontier/provenance/digest oracle, and fixed latency/resource gates. The earlier all-cold snapshot and 64-entry SGLang-trace hot tier remain closed no-go configurations.
7. Knowledge increment/boundary: authored-link substrate locality must be measured at the real retrieval-to-expansion boundary, not inferred from natural-language actions. This change is only canonical-public-data-code-integration/host-unit-correctness, performance_claim=false; it is not retrieval quality, serving, model, GPU or NPU evidence.

Stop condition: if the current canonical benchmark lacks a single pre-expansion ranked-document seam or trace publication would require answer/supporting-fact access, stop with 0 code. Hardware: CPU/filesystem only; expected implementation and host validation 30–45 minutes.

[ShuhaoZhangTony]

本轮组合审核点评

当前判断： 历史结果看起来有潜力，但当前 provenance 不足，先按重建证据处理。

我认可的证据： 我认可 link-aware retrieval 是清晰问题，也保留已有 recall/MRR 报告作为线索。

目前不认可的部分： 我不接受 baseline latency 为零或缺少 immutable query/data/run manifest 的比较。

下一步最小动作： 先固定数据版本、query 集、强基线配置和逐查询原始输出，再复跑一次 matched retrieval。

停止条件： 若无法重建历史输入或强基线后收益不再成立，就撤回倍数型效果表述。

[ShuhaoZhangTony]

2026-08-25 canonical native-Okapi frontier/run preregistration

This comment freezes the first complete canonical evidence cell before execution. It does not claim a production or Lucene result.

Immutable inputs

• HotpotQA distractor validation parquet: Hugging Face repo hotpotqa/hotpot_qa, commit 1908d6afbbead072334abe2965f91bd2709910ab, 7,405 queries, 27,452,575 bytes, SHA-256 c20b638ca82b21d04fe12e14ff417ad05153d4d215a65de54497fca4e972f7c6.
• Authored link graph: data/hotpotqa_authored_links.jsonl, SHA-256 5d335ba19aa1f7aafaf13682a75e5c48d4c528ca3d243c8e94422cefaf26b355.
• Producer commit: 0bcdf9a5e83d4276903554ab3dd2b342d68109b9 on feat/wiki-link-native-bm25-frontier-evidence-20260825 (the run must fail closed if the worktree is dirty).
• Query order: source parquet row order; first-stage cutoff 100.

Frozen first-stage contract

• Backend label: native-okapi (explicitly not Pyserini/Lucene).
• In-process Okapi BM25; k1=0.9, b=0.4; Unicode word-character tokens of minimum length 2, lowercased; binary query-term frequency; document-ID ascending tie break.
• Seed 42 (ranking is deterministic and seed-independent). Frontier trace contains request identity plus pre-expansion doc IDs/ranks/scores, no questions, answers, qrels, or label-derived fields.

Matched arms and fixed treatment

• B0: native-Okapi first-stage ranking, no link expansion.
• B1: same frozen first-stage ranking plus the authored bidirectional graph; alpha=0.3, max_expansion=3, protected_prefix=10, fusion_mode=raw-score.
• Metrics: nDCG@10, Recall@100, MRR@10, per-query paired effects, candidate-flow/reachability strata, and lookup latency. Paired bootstrap: 10,000 samples, seed 2027.

Admission and stopping rules

1. Exactly 7,405 unique frontier events and one valid manifest; dataset, graph, producer-config, and event hashes must verify.
2. B0 and B1 must share the exact first-stage ranking; no label-conditioned tuning or post-run parameter changes.
3. Any stale schema, duplicate scope, dirty producer, checksum mismatch, missing output, or non-finite score invalidates the cell.
4. This cell may support a qualified native-Okapi offline retrieval statement only. It cannot replace the separate Lucene/Pyserini gate and cannot be described as serving or production evidence.
5. Report full distributions and negative results. If B1 does not improve a preregistered metric or materially regresses another, retain the result and stop mechanism escalation rather than tuning around it.

[ShuhaoZhangTony]

2026-08-25 canonical native-Okapi cell — completed and merged

PR #12 is merged to main at 4d2ca17c3f9470652a156ded6b74c5e467e64cc2.

Gate outcome

• Full HotpotQA distractor validation set: 7,405 queries, 66,581 corpus documents, 14,810 relevance judgments.
• Frontier validator: PASS, schema v1, 7,405 events, one scope, no label fields/performance claim.
• Frontier SHA-256: c1fa59eacc239585ba850a631d6070ed5e7775966e80783bb1c3873275894fb4; events SHA-256: 58db28a910252e9e49fc9db6bef81ee932943e1762f680519b62de6dc813e579.
• Result SHA-256: 0c753c58388a31db4ea244d499790344ff371ae7a3cd8df7807e2a445a8b1fdf.
• Raw artifacts: /data/experiments/wiki-link/native_okapi_frontier_20260825/; checked-in manifest: results/native_okapi_frontier_20260825.manifest.json.

Frozen matched result

• B0 native Okapi: nDCG@10 0.6454, Recall@100 0.8936, MRR@10 0.7802.
• B1 authored links (alpha=.3, m=3, protected prefix 10): nDCG@10 0.6454, Recall@100 0.9143, MRR@10 0.7802.
• Paired Recall@100 mean delta: +0.020729; 95% paired-query bootstrap CI [+0.018366, +0.023160].
• Query distribution: 320 improved / 13 harmed / 7,072 unchanged. Candidate-flow recall identity max error: 0.
• Structural opportunity: 1,435 / 1,576 missed gold documents are one-hop reachable (0.910533).

Verification and evidence boundary

• Focused suite: 59 passed. Full suite with pinned parquet mounted read-only: 200 passed, 1 skipped.
• Admitted: canonical label-free frontier trace and qualified native-Okapi offline retrieval quality.
• Not admitted: latency (legacy schema records B0 as zero and B1 expansion-only), Lucene/Pyserini equivalence, serving, or production claims.

The exact next blocker is now narrower: repair the matched latency contract and execute the separate strict Lucene/Pyserini gate; the previous “no complete canonical frontier” blocker is closed.