Groth16 prover: memory scoping + MSM acceleration + prover options - #628
Open
OBrezhniev wants to merge 127 commits into
Open
Groth16 prover: memory scoping + MSM acceleration + prover options#628OBrezhniev wants to merge 127 commits into
OBrezhniev wants to merge 127 commits into
Conversation
# Conflicts: # build/snarkjs.js # build/snarkjs.min.js # src/groth16_prove.js
…y needed. Changed a few anonymous functions to named for easier profiling. Smaller chunks in joinABC to split work in most cases, and pass buffer ownership to worker threads there.
…ly when they are needed (bfj, ejs). And don't use bfj module for small json files (proofs, public signals). Comment out vm module and manual garbage collection in zkey_new.js.
…Switch between different buildABC implementations through options. Debug & logging in groth16_prove. Rebuild.
Picks up the 2-phase termination and WorkerSlot identity model from ffjavascript so all buildABC modes (js/wasm/wasm1) run reliably without intermittent "Worker terminated unexpectedly" failures or hangs.
Picks up ffjavascript console.log cleanup (engine_fft, engine_multiexp, threadman) via rebuilt bundles. No logic changes in snarkjs src. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Picks up the worker-side in-place reversePermutation (no WASM memory growth, zero-copy) via the inlined ffjavascript in the IIFE bundle. No snarkjs src changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…script Picks up the ffjavascript fix that stops pairingEq from detaching caller-owned G1.g/G2.g buffers. Restores the full prove/verify pipeline: snarkjs test suite now 49/49 passing. No snarkjs src changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ding Picks up the fastfile node:fs->fs.constants fix and the ffjavascript bn128 prebuilt-wasm loader fix (atob/arrayBuffer) so the browser builds run in a real browser again. snarkjs browser test suite (browser_tests) now passes: full setup/prove/verify in headless Chrome on both the IIFE and ESM builds. No snarkjs src or rollup config changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…pt; drop dead taskmanager
- misc.getRandomBytes / sha256digest: drop `process.browser`. Prefer the Node
crypto module (createHash, randomFillSync -- no per-call size limit), fall back
to Web Crypto (getRandomValues chunked to 65536 bytes; subtle.digest on the
view, not data.buffer, so subarray byteOffset/byteLength is respected).
- askEntropy: use the browser prompt only when a real DOM window exists
(typeof window / window.prompt), not !process.browser -- "not Node" is not the
same as "browser" (Bun/Deno/edge/SES have neither).
- Delete src/taskmanager.js: unused dead code, and the only place using
`new Worker(code, {eval:true})` + require() codegen, which trips CSP/eval
scanners.
Rebuilt browser bundles. (package.json fastfile dep left as-is; separate.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…stubs Rebuilt IIFE bundles after the ffjavascript (os/crypto) and fastfile (fs/constants) "browser" field additions. No snarkjs src changes. Browser e2e (IIFE + ESM) passes; bundles have no Node-builtin leaks. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Picks up ffjavascript's vendored, statically-imported prebuilt wasm (no dynamic import of wasmcurves, no gzip decode). No snarkjs src changes. Browser e2e (IIFE + ESM) and tutorial e2e pass; node suite 49/49. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… from ffjavascript Picks up ffjavascript's lazy getWorkerSource (no Blob/btoa at import) and the base64 decoder that prefers Buffer/atob with a pure-JS SES fallback. No snarkjs src changes; browser e2e + node suite pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The five base-point sections (A/B1/B2/C/H) were each read whole into a buffer and then sliced per worker chunk -- two full copies of every section, and all five held in RAM concurrently (the calc promises run in parallel). Switch them to curve.G1/G2.multiExpAffineChunked with a per-section reader (mkSectionReader) that returns each chunk directly via fdZKey.readToBuffer. No full section buffer, no main-thread slice, and only a few chunks resident at a time. Using readToBuffer directly (rather than binFileUtils.readSection per chunk) also avoids amplifying readSection's per-call console.time logging by ~40x/section. Effect scales with circuit size: - authV3 (29 MB zkey, 19 MB bases): peak RSS ~603 -> ~586 MB, time neutral. - sha256 (1.1 GB zkey, 733 MB bases): peak RSS ~3.85 -> ~3.41 GB (~12%), and a bit faster + lower variance (less GC pressure from the big transient allocs). Proof identical / verifies OK in both cases. Validated: snarkjs 49, ffjavascript 63, tutorial e2e (groth16/plonk/fflonk), authV3 + sha256 prove+verify. Bundles rebuilt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The default buildABC for circuits whose witness fits one pass was buildABCWASM1: a single worker that loads ALL coefficients (e.g. 357 MB) + witness + the three output buffers (~621 MB) and never shrinks (WASM memory can't), so that ~681 MB stayed resident for the whole prove, and the build ran single-threaded. Replace it with a streaming build: - buildABCStream: still a SINGLE full-witness pass (so each disjoint output range is computed completely -> no batchAdd/joinABC merge, unlike multi-threaded buildABC), but the domain is split into nChunks output ranges processed with bounded in-flight. Each task holds only witness + one coeff chunk + one output chunk, so a worker's high-water is ~witness+chunk instead of the whole thing. - pickStreamParams: adaptive. Each busy worker's WASM memory persists, so the floor buildABC leaves behind is ~maxInFlight x perWorker. It sizes nChunks so a worker holds ~2x witness, derives maxInFlight from a worker-memory floor budget (default 256 MB), and sets nChunks to a few per BUSY worker (not full concurrency -- that would re-copy the witness per chunk for nothing at low parallelism). Small circuits -> full parallelism; large -> bounded. Tunable via options.buildABCFloorBudget / buildABCnChunks / buildABCmaxInFlight. - Default path uses streaming whenever the witness fits a single pass (all normal circuits); multi-threaded buildABC stays as the witness-too-big fallback. buildABCWASM1 is kept for the explicit "wasm1" option. Measured (sha256, 1.1GB zkey, vs the old wasm1 default 9.11s / 3354 MB): - default 256 MB budget (n9/k2): ~8.5s / ~3.05 GB -> ~7% faster AND ~9% less peak - memory-first (<=192 MB budget, k1): ~9.3s / ~2.9 GB -> -13% peak, +2% time - raising the budget is counterproductive (more memory, eventually slower from worker oversubscription against the concurrent multiexps); the knob is useful only downward. authV3 (small) picks n45/k15 -> unchanged (~1.08s). All verify OK. Validated: snarkjs 49, tutorial e2e (groth16/plonk/fflonk), authV3 + sha256 prove+verify. Bundles rebuilt. ffjavascript untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Picks up the binfileutils MAX_BUFFER_SIZE and ffjavascript BigBuffer PAGE_SIZE cleanup (dead `Buffer.constants` probe replaced with an explicit `1 << 30`). The IIFE and browser-ESM bundles inline those deps; main.cjs/cli.cjs import them externally and are unaffected. No behaviour change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The IFFT->applyKey->FFT pipeline drops each FFT/IFFT input immediately (buffX_T is nulled; buffXodd goes out of scope), so pass consume=true to ffjavascript's fft/ifft and skip its defensive full-input copy. The 3 FFT inputs are flat Uint8Arrays (from batchApplyKey) and are consumed; the 3 IFFT inputs are BigBuffers (from buildABC) and still flatten as before. sha256: ~100-300 MB lower peak RSS, time within noise (the FFT copies overlap the concurrent multiexps, so the win is modest). authV3 + sha256 verify OK; snarkjs 49, ffjavascript 64, tutorial e2e all pass. Bundles rebuilt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When the domain fits under BigBuffer's 1 GiB page, allocate the A/B/C outputs as flat Uint8Arrays instead of BigBuffers. The downstream IFFT (consume=true) can then take them in place and skip its defensive full-input copy -- previously a BigBuffer input forced a flatten-copy. Larger domains stay paged BigBuffers (the IFFT flattens those as before). ~2.5% faster end-to-end on sha256 (8.59s -> 8.37s, quiet machine, the whole run distribution shifts down); peak RSS unchanged (the consumed copies are early in the pipeline, not at the peak). authV3 + sha256 verify OK. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
buildABCWASM1 (single worker, one full-witness pass, all coeffs at once) is exactly buildABCStream with nChunks=1 -- and the streaming default strictly dominates it (lower, bounded worker memory + tunable parallelism). Delete the ~150-line function and route the explicit "wasm1" option to buildABCStream(..., 1, 1), byte-identical. The other variants stay: "wasm" (multi-threaded, the witness-too-big-for-one-pass fallback that splits the witness across passes) and "js" (pure-JS element-at-a-time, zero bulk wasm allocation -- the universal fallback for arrays beyond wasm's 32-bit limit). Behaviour-preserving; default/wasm1/js/wasm all verify OK. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
r1csInfo prints header counts (wires, constraints, inputs, labels, outputs) but
called readR1cs() with defaults, which loads the entire constraints section (and
the wire-to-label map) -- reading the whole, potentially many-GB, .r1cs file just
to print a few numbers. On a 12 GB r1cs this took minutes and gigabytes of RAM.
Pass {loadConstraints: false, loadMap: false}: the counts all come from section 1
(the header). Same output, now near-instant (12 GB r1cs: ~minutes -> 0.17 s,
~79 MB RSS). r1csInfo never touches cir.constraints, and no caller uses the
returned value's constraints.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
options.msmBatching selects the ffjavascript MSM batching mode and is threaded to all five multiexps (A, B1, B2, C, H): "auto" (default) batch-affine only for cache-friendly chunk sizes; "enabled" always batch (best for small/medium circuits); "disabled" plain multiexp (lowest memory; best for very large circuits). Invalid values throw.
wasmcurves b83021f: 99.5% lines / 100% functions (was 78.8% / 78.3%), 151 tests. Fixed isPrime never reaching Miller-Rabin (build_f1m silently skipped generating _sqrt/_isSquare for any prime outside the hardcoded bn254/bls12-381/mnt6753 list); removed three broken wasmsnark leftovers (build_mulacc, build_mem, build_testg1); revived the excluded 27-test mnt6753 suite. Chain: ffjavascript 3cc2f7a, binfileutils 824e9f1, r1csfile f0133fd. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
…Node 20 dispatch) Fixes the failing browser CI job: in Chrome the Range probe for URL-read files is answered 206 from the browser's HTTP cache while the st static server ignores Range, so the reader aborted with 'file changed while reading' during verifyFromR1cs. fastfile dba4d82 degrades to full buffering when a mid-session 200 carries an unchanged strong validator. Also via ffjavascript 6ba342f: worker teardown is hard-terminated portably (a Bun process could never exit after proving -- the residue behind snarkjs#490/#533), and dispatching a detached transfer buffer rejects on every Node version (Node 20 posted silently and hung). Chain: binfileutils 9a54502, r1csfile 85827cd. Bundles rebuilt; browser test suite passes locally. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
Local installs ran with --allow-git=root, which silently omits transitive git dependencies (ffjavascript -> wasmcurves) from the lock; CI's npm ci then fails with 'Missing: wasmcurves from lock file'. Regenerated with --allow-git=all. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
…ory-scoping) Port of the feature/esm-tooling migration (41a8ca3) onto the current branch: Rollup -> Vite (node/cli/browser/iife builds), Mocha -> Vitest (node-esm + browser-chromium via Playwright), flat eslint.config.js. Everything since the original fork point is preserved and converted: - All 14 post-fork test suites (coverage program, standards vectors, optimization-levels matrix, http streaming, buildABC gap, ptau/zkey tools) converted: before/after -> beforeAll/afterAll, this.timeout dropped (config testTimeout/hookTimeout 600s). Node assert imports kept. - node-esm runs test files sequentially (fileParallelism: false): every suite builds a curve with a full worker pool, and parallel files oversubscribe the CPU badly enough that the large-domain FFT test starves past 600s. Sequential, the whole suite is 246 tests in ~45s. - CLI banner is the migration's portable "#!/usr/bin/env node"; all globalThis.gc() call sites are guarded, so dropping --expose-gc is safe (profile with `node --expose-gc build/cli.cjs` as before). - Indent autofix reindented the fd-leak `try { ... } finally` bodies that were deliberately left unindented to minimize those diffs (whitespace-only); eslint caughtErrors: none matches the absorb-close catch(e){} style. - CI keeps OUR three-job workflow (3-OS matrix, hardhat verifier contracts, browser_tests bundle harness) and adds a vitest browser-chromium job; actions bumped to checkout@v6 / setup-node@v6. - Re-pinned all four siblings to their migrated SHAs: binfileutils f71fb82, ffjavascript 4ac1cba, fastfile e7eb0f0, r1csfile 09b93c2. vitest family ^4.1.11 (GHSA-p63j-vcc4-9vmv); postcss/brace-expansion overrides (npm audit clean). Mocha config block removed. 246 node tests + 10 browser tests (Chromium) pass; lint clean; all bundles rebuilt with vite (snarkjs.js dropped, only the umd-referenced snarkjs.min.js is built); CLI r1cs-info and CJS require smoke-verified. (cherry picked from commit 41a8ca3) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
- Re-pin ffjavascript 2c20f30 (sequential vitest files — CI starvation fix), fastfile 748772b, binfileutils f8aedac, r1csfile f5ae083. - browser.esm.js externalizes only ffjavascript again, bundling binfileutils/ r1csfile/fastfile, matching the old rollup build: the browser_tests harness (and any consumer with the documented importmap) only maps ffjavascript, so the wider externalization broke bare-specifier resolution. - browser_tests point at snarkjs.min.js (the unminified snarkjs.js IIFE is no longer built). 246 node tests, 10 vitest browser tests, and the puppeteer bundle harness (IIFE + ESM against a real ceremony ptau) all pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
ffjavascript a6abbfe (lockdown harness explicit exit, execFileSync timeouts, CI timeout-minutes), fastfile e74f95d, binfileutils 9ee4f27, r1csfile 0a53eb0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
…ade) wasmcurves c380d6c -> ffjavascript 5475db8, fastfile 38ffb38, binfileutils 7127862, r1csfile 0235e34. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
wasmcurves 13d4acc -> ffjavascript 76fc09a; fastfile ad59e79; binfileutils c69440f; r1csfile bf6933b. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
ffjavascript f16db3c (zombie worker threads can no longer hold the process open after terminate) -> fastfile 87d38de, binfileutils 6c2616f, r1csfile e0bb6f8. Browser bundles vendored from ffjavascript rebuilt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
…not main Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
…urces)
groth16.prove / groth16.fullProve (proverOptions) and wtns.calculate
(options) accept persistentCache: true | {blockSize, maxBytes, dbName}.
When the zkey (or circuit wasm) is an absolute http(s) URL string or an
explicit {type: "http"} descriptor, the open is routed through fastfile's
IndexedDB block cache: the first proving session populates it, later
sessions read the key locally -- the warm-start e2e proves the second
prove touches the network for the open probe only. Local paths, mem/bigMem
descriptors and open fds pass through untouched, and in Node (no
IndexedDB) the option is a safe no-op.
Re-pinned fastfile to the persistentCache prototype (4ee05be, the
feature/direct_rw_optimization line) and binfileutils to a066c87 -- which
includes the essential rebuild of its browser bundle: that bundle INLINES
fastfile's browser build, so a fastfile re-pin without a binfileutils
rebuild ships the stale copy and the option is silently ignored (found the
hard way: the warm-start test saw 11 fetches instead of 1). snarkjs's own
browser bundles inline binfileutils in turn and are rebuilt here too.
Tests: browser warm-start e2e (cold prove populates the cache, warm prove
= 1 probe request, both proofs verify), Node no-op prove over the http
test server, and unit coverage of the source-mapping helper.
249 node + 11 browser tests pass; lint clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
… bundles Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
…uild bundles Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
… rebuild bundles Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
…; rebuild bundles Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
…dles ffjavascript 4f77fd9 (the root cause of the random CI hangs) -> fastfile 52b1732, binfileutils 1587193, r1csfile b53d52d. Browser bundles vendored from ffjavascript rebuilt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
…; rebuild bundles ffjavascript 8430220 (worker idle request crossing a task dispatch was torn down with the task aboard, hanging the await -- the residual rare CI timeout) -> fastfile 653bce6, binfileutils b04e62f, r1csfile 4d33f21. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
…out); rebuild bundles Range requests now retry transient failures (network drops, 5xx/429, stalls via a 10s watchdog) with exponential backoff before propagating; permanent errors (4xx, changed validator) fail fast. Browser proving over unstable connections survives intermittent faults within a single prove() call. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
…fy-path fixes) Audit results for the groth16 verify path: - Safe by construction: every ffjavascript sync-op region is await-free (atomic on the JS thread) with finally-guarded pointer reset; curve point ops use fixed scratch slots (no per-call wasm allocation); miller loops run in isolated worker memories. - Two reachable races found and fixed in ffjavascript 9f3109d: terminate() during an in-flight task dropped its deferred (hung concurrent verifiers of the shared cached curve -- now rejected explicitly), and concurrent first-time curve builds created two worker pools and returned different objects (the in-flight build promise is now cached). New stress suite (committed fixtures: deterministic zkey + one saved proof): 32 concurrent verifies on the shared curve; interleaved valid/corrupt-proof/out-of-field-signal verifies stay individually correct with no wedged shared state; 200 sequential verifies hold the wasm bump allocator exactly constant and RSS flat. Re-pins: ffjavascript 9f3109d -> fastfile 3014b8c, binfileutils 3901abd, r1csfile 81a9cbf; bundles rebuilt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
Static audit of the prove path found no new shared-state hazards beyond the verify audit: the prove-side engine modules (multiexp, fft, applykey, batchconvert) do no main-thread wasm allocation at all -- everything runs as worker tasks with isolated memories -- and the only main-thread allocations anywhere are the construction-time fixed scalar/point scratch slots, used only inside synchronous (await-free, hence atomic) code. tm.resetMemory() would corrupt those fixed slots if called, but has zero callers in ffjavascript or snarkjs. Empirical coverage: 8 concurrent provers + 24 concurrent verifiers on the shared cached curve (every fresh proof verifies, every verify correct, wasm bump allocator exactly unchanged after the storm); interleaved prove/valid-verify/failing-verify rounds stay individually correct; two concurrent provers produce distinct, independently valid proofs (identical pi_a would flag shared randomness/scratch contamination). 8x clean under the 2-CPU contention loop. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
Static audit extended to PLONK and FFLONK: Keccak256 Fiat-Shamir transcripts are per-call locals; the Polynomial/Evaluations machinery holds no module-scope state; both verifiers deep-copy their inputs via unstringifyBigInts, so fflonk_verify's vk.domainSize write mutates a private copy, never the caller's vk object. Heavy work runs as worker tasks; the only main-thread wasm use remains the fixed sync scratch slots. New suite enforces it empirically: a cross-protocol storm (plonk + fflonk provers, plonk/fflonk/groth16 verifiers all interleaving in one ThreadManager queue) with every result individually correct and the wasm bump allocator exactly unchanged; concurrent same-protocol provers produce distinct valid proofs (shared blinding randomness would collide); 10 concurrent fflonk verifies on ONE shared vk object leave it byte-identical; corrupt plonk/fflonk proofs verify false amid valid concurrent traffic. 6x clean under the 2-CPU contention loop; full suite 256 green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
Extend the groth16-setup memory-scoping approach (1d5a433) to the four remaining big-memory pipelines: release every large buffer at its last consumer instead of at function exit, with gc checkpoints after each release cluster. plonk setup: r1cs constraints dropped after processing; additions after section 3; constraint list + lagrange points after sigma; sigma slices, aparence maps and S1-S3 released inside writeSigma as they are consumed (counts captured up front for the header). plonk prove: witness + wire maps dropped once the wire buffers are built; buffers.B/C after computeZ (A after its round-3 public-input reads); all 13 extended evaluations (A/B/C/Z/Q*/Sigma*/Lagrange) released right after the T evaluations loop -- before the T iffts and MSMs; T/Tz staging dropped as T1-T3 are split; Q*/Sigma3/T1-T3 after computeR, R/A/B/C/Sigma1/Sigma2 after computeWxi, Z after computeWxiw. fflonk setup: constraint/addition arrays released after their last section write; Q and sigma extended evaluations dropped right after being written (they are only stored in the file); the 8 component polynomials dropped once C0 is assembled; PTau + C0 after the C0 commitment. fflonk prove: the existing round-boundary delete blocks moved to the true last-use points inside the rounds (selector evaluations after the T0 loop, wire buffers after computeZ, lagrange after computeT1, all extended evaluations after computeT2), plus C0/C1/C2 folded-into-L and F released inside round 5 instead of after it. Peak RSS on authV3 (441k constraints, /usr/bin/time -v), before -> after: plonk setup 3.43 GB -> 2.92 GB (-15%) prove 3.62 GB -> 3.08 GB (-15%) fflonk setup 4.80 GB -> 4.74 GB prove 6.32 GB -> 5.78 GB (-9%) Wall times unchanged (plonk prove 2:38 -> 2:34). Setup outputs verified byte-identical to pre-change baselines for authV3 and the small test circuits; proofs verify. Full suite green (267 passed). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
OBrezhniev
force-pushed
the
feature/memory-scoping
branch
from
August 29, 2026 01:06
1105c67 to
0aa5ee9
Compare
The MPC streaming loops (applyKeyToSection, ptau processSection /
hashSection) allocate a few MB of read/apply/hash buffers per chunk that
are all garbage one iteration later, but v8 collects them lazily, so RSS
grew with the section size; and the chunk buffers transferred to the
worker pool piled up as external memory in the worker isolates, whose
collectors never fire while idle (fixed in ffjavascript cc88563 by
transferring the input buffers back, re-pinned here).
- mpc_applykey / powersoftau_contribute: guarded gc checkpoints inside
the chunk loops; the challenge hash pass reads 4 MB chunks (was 16 MB)
since it only feeds the hasher.
- cli.js: the CLI is node-only, so it now exposes a gc itself (via
v8.setFlagsFromString + vm) when node was started without --expose-gc,
making every guarded checkpoint in the library effective for plain
`snarkjs <cmd>` runs. Library and browser consumers are unaffected.
- re-pin siblings for the ffjavascript worker fix: ffjavascript cc88563,
fastfile 011644e, binfileutils 63e62fe, r1csfile 29af5e0.
Measured with plain node (no flags), /usr/bin/time -v peak RSS:
ptau contribute p20: 1431 MB -> 841 MB, 4:07 -> 2:51
zkey contribute (1.1 GB g16 zkey): 756 MB -> 539 MB, 2:25 -> 1:16
ptau new p20: 292 MB (unchanged; already at the
curve/worker floor, like g16/plonk/
fflonk setup baselines)
Outputs validated: contributed ptau passes `powersoftau verify`; full
snarkjs suite (267) and ffjavascript suite (215) green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
Drop the -memlog CLI option, the memoryLogging prover option and the memUsage/monitorMemoryUsage helpers from groth16Prove, plus their fullprocess test. Peak-RSS measurement is done externally (/usr/bin/time -v) and the memory-scoping work this instrumented is landed. Bundles rebuilt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
…tentCache
fastfile now applies the caller's cacheSize/pageSize hints as fallbacks
to descriptor-object sources too (fastfile 1514b2a via binfileutils
baed18f), so {type: "http"|"file"} descriptors passed as zkey/wtns/wasm
sources get the same tuned page cache as plain path/URL strings, with
explicit fields winning. mem/bigMem descriptors keep their identity.
With that, the separate options.persistentCache plumbing in
groth16Prove and wtnsCalculate is redundant: pass the fastfile
descriptor directly instead --
{ type: "http", url, persistentCache: true | {blockSize, maxBytes, dbName} }
withPersistentCache is removed from misc.js and the browser warm-start
test now exercises the descriptor form end to end.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
… cache)
The IndexedDB warm-start field on {type: "http"} sources is now
`cache: true | {blockSize, maxBytes, dbName}` (fastfile 83ccfe5 via
binfileutils 795f9c1); same shapes and behavior, renamed to leave room
for a future injected cache implementation. Comments and the warm-start
tests updated; the node no-op test now exercises the descriptor form.
Bundles rebuilt.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
…t artifacts test/__screenshots__ (browser-mode failure captures) and .vitest-attachments were sweepable by git add; one screenshot slipped into a41c924. Removed and both directories ignored. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
The repo has always been LF (verified: no CR bytes in any tracked blob), but without an EOL policy a clone with core.autocrlf=true materializes CRLF in the working tree. text=auto eol=lf makes every checkout LF; binary fixtures are exempted explicitly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
…l-Rust already Rolldown bundles, oxc transforms and minifies; no esbuild/terser/rollup anywhere in the tree. rimraf was the last redundant dev dependency. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Groth16 prover: memory scoping + MSM acceleration + msmBatching option
Summary
Companion to the
ffjavascript/wasmcurvesPRs (file-linked dependencies —land all three together). Two arcs:
Memory scoping — chunked zkey bases reads (streamed to workers instead of
whole-section buffers), adaptive streaming buildABC with per-chunk witness
gathering (the witness never enters WASM, so there is no witness size limit;
replaces the retired wasm1/multi-threaded variants and the oversized-witness
fallback), fft
consumewiring in the abc pipeline, explicit 1 GiBBigBuffer/binfileutils paging, fast
r1cs info(header-only read).MSM acceleration — exposes
options.msmBatching = "auto"|"enabled"|"disabled"on
groth16Prove, threaded to all five multiexps (A, B1, B2, C, H). Defaultautopicks the batch/endomorphism path per chunk size.Results (all proofs verify; suites pass):
Also included
browser_tests/bench.mjs: instrumented in-browser proving benchmark(prove wall, main-thread heap, renderer-tree RSS scoped to the launched
Chrome, in-page verification).
only reachable via ffjavascript's unused custom-
pluginspath) stubbed outof the single-file browser builds —
snarkjs.js5.50 → 3.81 MB,snarkjs.min.js756 → 573 KB (−24%).--memlog[=ms]ongroth16 prove/fullprove(API:memoryLogging):opt-in periodic heap/RSS/external logging, Node-gated, cleared on
completion with a final sample. Debug console output removed from the
prover (timers now only via the optional logger).
mocha 11, ejs 6); eslint surfaced and fixed a latent ReferenceError in
Polynomial.expXplus assorted dead code.the manifest resolves standalone; local dev uses uncommitted
file:overrides. Land the sibling PRs first, then re-pin here before merge.
Validation
49 passing; groth16 e2e (prove+verify) on authV3 and sha256 at every step;
CLI smoke tests on the built bundles; browser runs verify in-page.
🤖 Generated with Claude Code