perf(dflash): split-chain fast rollback with F32 SSM checkpoints (+29% decode)

[Amidwestnoob]

Summary

• add layer-split F32 SSM checkpoint capture and fast pure-chain rollback for DFLASH
• keep production tree verification fail-closed and reject sibling/non-chain geometry before execution
• add a CPU-only root-inclusive pure-chain guard regression test

Benchmark

Controlled dual-GPU A/B/A benchmark on two RTX 3090 24 GB cards using the same pinned models, prompt, generation length, power/thermal conditions, and layer split (--target-gpus 0,1 --target-layer-split 12,11). Each block used one discarded warmup and five measured runs.

Metric	Baseline	Candidate	Delta
Median decode throughput	22.76 tok/s	29.41 tok/s	+29.22%
Median wall time	24.895 s	22.560 s	-9.38%
Prompt processing	~95.4 tok/s	~95.4 tok/s	flat
Acceptance rate	42.4%	37.2%	-5.2 pp
Average commit	7.76	5.95	-1.81

All 15 measured runs produced byte-identical 256-token output:

1653a412811d3b87e26aab900e801b411fd68f9214b512ae984db6404b82204f

A1/A2 baseline drift was 0.48%. Maximum observed GPU temperature was 64 C.

Memory trade-off

The F32 SSM checkpoint storage adds exactly 1,774,190,592 bytes (~1.65 GiB) of persistent GPU memory in the tested split configuration. Both GPUs are required for the tested 27B configuration; a single RTX 3090 is not sufficient.

Safety boundary

• supports_tree_verify() remains unconditionally false in production.
• Only the validated root-inclusive pure-chain rollback path is enabled.
• Sibling, malformed-root, skipped-parent, cyclic/forward-parent, out-of-range, truncated, and null geometry is rejected before the execution sentinel.
• Fast rollback is opt-in through DFLASH_SPLIT_FAST_ROLLBACK and requires successful capture validation.
• Temporary evidence instrumentation is excluded from this commit.

Verification

• rebased onto current upstream main at 5e302cbb483819cd21e72f5dd8becaa609eca8cf
• preserved upstream fix(qwen35): route non-tree GDN capture through persist buffer (src[7]) #469 non-tree GDN persist-buffer routing through src[7]
• full Release CUDA 8.6 build and link: passed
• current upstream test suite: 15/15 passed, including CUDA and the pure-chain guard
• rebased-candidate dual-GPU validation: passed; 43/43 fast rollbacks succeeded with zero fallback
• rebased validation produced the same generated suffix hash as the controlled benchmark
• controlled A/B/A benchmark: 18/18 isolated processes passed, including warmups
• no OOM, CUDA, fallback, tree-path, output-parity, or thermal failures

Scope

Measured on the exact dual-RTX-3090 configuration above with a 1,024-token prompt and 256 generated tokens. No performance claim is made for single-GPU execution, sibling-tree verification, other models, or other hardware.

[cubic-dev-ai]

All reported issues were addressed across 13 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[davide221]

@Amidwestnoob thanks for the contribution, please review and fix cubic reviews

[Amidwestnoob]

@howard0su When you have a chance, could you please review this? Cubic passes and all four review threads are resolved. The latest follow-up gates capture logging behind the existing self-test/diagnostic flags; the Release CUDA build completed and all 15 tests passed locally.

[dusterbloom]

@Amidwestnoob I did run it and there is still little gain that can be squeezed.

See my PR feel free to diff it and cherry pick the commits I added on top of your good PR 506

[Amidwestnoob]

Added 263cf60 (perf(qwen35): batch recurrent snapshot synchronization), adapted from dusterbloom's bb76966 while preserving the later split-capture diagnostic gate.

Verification on the exact PR head lineage:

• Built dflash_common, test_recurrent_snapshot, test_qwen35_split_tree_guard, test_dflash, test_generate, dflash_server, backend_ipc_daemon, and test_server_unit.
• Focused CTest: recurrent_snapshot, qwen35_split_tree_guard, and server_unit — 3/3 passed.
• Actual single-GPU and dual-GPU layer-split server health/chat flows passed.
• Three measured runs per build after warmup, same RTX 3090/model/workload:
  • snapshot + restore median: 1.79 → 1.12 ms/step (37.43% reduction)
  • decode median: 40.43 → 40.61 tok/s
• All six measured baseline/patched output streams were byte-identical (25ab6ff1b0ce3b3e7ed9bd6a453e198de0ee0df9813630775c30efe228de8183).

The bounded divergence check also found a pre-existing target-vs-DFlash mismatch at generated token 111 in both legacy and fast modes, before their restore-path outputs differ. That finding is not introduced by this batching commit and should be investigated separately in verifier/acceptance/state-selection semantics.

[cubic-dev-ai]

All reported issues were addressed across 9 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[Amidwestnoob]

Rebased PR #506 onto current upstream main after #515 merged.

• old head: 263cf60f2431a8374ab074a1e86e5b9e886e8a54
• new base: f258c4a6e66c76f6a26fa7462dc9716017634003
• new head: 76e85a8657e4a57285846b9a835eb39c242738e2
• range-diff: four-commit series preserved; duplicate single-GPU pieces already supplied by perf(qwen35): add opt-in F32 rollback checkpoints for single-GPU inference #515 were dropped
• perf(qwen35): add opt-in F32 rollback checkpoints for single-GPU inference #515 remains the source of truth for single-GPU rollback policy (DFLASH_SINGLE_CHAIN_*); this PR retains split-chain rollback (DFLASH_SPLIT_FAST_ROLLBACK) and recurrent snapshot batching

Post-rebase validation on RTX 3090 / sm_86:

• Release CUDA build: pass
• full configured CTest suite: 20/20 pass
• single-GPU default vs F32 checkpoint smoke: pass; identical generated-token SHA-256 2ece303dc333184ac1bdaf7495e02655cbef57926d28606fa4b225ab67ab0b1c
• dual-GPU split-chain smoke (12,11, 256 tokens): pass
• dual-GPU generated-token SHA-256: 1653a412811d3b87e26aab900e801b411fd68f9214b512ae984db6404b82204f, exactly matching the previously validated pre-rebase candidate
• no CUDA errors, OOMs, NaNs, invalid tokens, or rollback fallback failures detected

The force-update used an explicit lease against the old head and was read back from GitHub.

[cubic-dev-ai]

1 issue found across 4 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="server/src/server/server_main.cpp">

<violation number="1" location="server/src/server/server_main.cpp:1027">
P3: Startup can report `split_rollback = ON` while no fast rollback is usable: the displayed value is only the env request, but runtime also requires local storage and successful capture validation. Label this as requested (or report effective status after validation) so operators do not mistake a fail-closed fallback for enabled acceleration.</violation>
</file>

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

P3: Startup can report split_rollback = ON while no fast rollback is usable: the displayed value is only the env request, but runtime also requires local storage and successful capture validation. Label this as requested (or report effective status after validation) so operators do not mistake a fail-closed fallback for enabled acceleration.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/server/server_main.cpp, line 1027:

<comment>Startup can report `split_rollback = ON` while no fast rollback is usable: the displayed value is only the env request, but runtime also requires local storage and successful capture validation. Label this as requested (or report effective status after validation) so operators do not mistake a fail-closed fallback for enabled acceleration.</comment>

<file context>
@@ -980,6 +1022,10 @@ int main(int argc, char ** argv) {
     std::fprintf(stderr, "[server] │  fast_rollback   = %s\n", bargs.fast_rollback ? "ON" : "off");
+    if (bargs.device.is_layer_split()) {
+        std::fprintf(stderr, "[server] │  split_rollback  = %s\n",
+                     split_chain_fast_rollback_enabled() ? "ON" : "off");
+    }
     std::fprintf(stderr, "[server] │  ddtree_budget   = %d\n", bargs.ddtree_budget);
</file context>

Suggested change

	split_chain_fast_rollback_enabled() ? "ON" : "off");
	split_chain_fast_rollback_enabled() ? "requested" : "off");