Community benchmark: we evaluated MemOS (OSS v2.0.33) on a Chinese long-context causal-reasoning benchmark — numbers + invite to correct

[FuRongJun-1999]

Context

hive-memory-bench is a Chinese long-context memory benchmark (a 23-chapter, ~61k-character fiction corpus + 184 causal-reasoning questions). In our "memory–generation separation" design, every memory system is attached to the same generator (deepseek-flash), so the only variable is what the memory layer retrieves.

The comparison covers six systems: OpenViking, Mem0, MemOS, Hindsight, a naive BM25 baseline, and dsh-memory (灵枢). Full disclosure: dsh-memory is our own project — which is one reason every raw reading is public and recomputable, and why we're posting this for review rather than as a verdict.

What we measured (two cut points: A = ch.1–8, B = ch.1–16)

• L1 verbatim rate — share of returned segments locatable verbatim (≥20-char exact overlap) in the source material.
• L2 verbatim ground-sentence retention — same test, restricted to "load-bearing" sentences (identified by our answer key).
• L3 causal layer — the retrieval output is given to the generator as the only context; we measure whether the answer's citations can be located verbatim in the source (evidence-chain checkability), and how many ground sentences the answer actually uses.

MemOS's results (OSS v2.0.33)

	cut A	cut B
L1 verbatim rate	40.0%	45.0%
L2 ground-sentence retention	15.8%	10.0%
L3 citations verbatim-locatable	2/120 (2%)	1/116 (1%)
L3 ground-sentence usage	2/19	1/20
write time	304 s	599 s

Related numbers from the same batch: in a zero-material control (no material at all, same generator, same judge batch), the baseline scored 67.9% vs MemOS's 56.9% (net material contribution −11.0pp, p=0.018); and in the "verbatim basis" column, MemOS's answers scored 2%/1% locatable citations.

What we think this says — and why it's architectural, not a score complaint: MemOS's retrieval knows where to look (its chunk-level hits point at the right chapters), but it delivers rewritten fact statements. The generator cites those statements faithfully; those citations then do not exist in the source text — so the answer's evidence chain can't be checked against the original corpus. (2% ≠ "wrong answer": the reasoning content may still be good; the metric measures evidence checkability.)

Full tables: docs/记忆系统对比_v1.0.md · every cell machine-readable: systems/readings.json

Caveats about our setup — please correct us

• We tested the OSS package v2.0.33 with the Neo4j + Qdrant backends; extraction LLM fixed at deepseek-flash; retrieval k=10, no source-echo mode for these tables.
• If there is a recommended configuration, prompt set, or newer release that changes extraction behavior, tell us and we'll rerun.

Invitation to verify

If anything above is misconfigured, tell us and we will rerun and update the public readings with credit. The method is re-runnable on anyone's corpus: systems/三轴诊断.py takes (your material + a system's retrieval output + your ground-sentence list) and produces the same three-axis readings.

Our own system is in the same comparison and judged by the same numbers: dsh-memory (灵枢).