zstd: speed up better encoder hashing, primes in registers on arm64

[lizthegrey]

Speeds up SpeedBetterCompression encoding: +6–8% on arm64 and +3% on amd64, with byte-identical output.

The motivating workload is a compress-and-upload service on arm64 (Graviton) that encodes 4 MiB frames with a 4 MiB window at SpeedBetterCompression.

Changes

betterFastEncoder and betterFastEncoderDict hash every position with two 64-bit multiplicative primes.

• arm64: each prime takes 4 instructions (MOVD + 3×MOVK) to build, and the compiler rebuilds it on every use, including on every index-loop iteration. betterHashPrimes() now loads both primes once per Encode from a package var, so they stay in registers.
• Other architectures: betterHashPrimes() returns the constants. amd64 builds a prime in one MOVQ $imm64, and keeping both live across the main loop spills. The choice is a single constant, hashPrimesInRegs = runtime.GOARCH == "arm64" in hash.go, and the compiler removes the unused branch.
• Index loops, all architectures: the loop bound is computed once (for end := s - 1; index0 < end; index0 += 2), and the short-table store now comes before the long-table update. Together these remove per-iteration spills on amd64.

Why output is unchanged

• betterHashL/betterHashS compute exactly what hashLen computes for these table sizes. A new table test checks them against hashLen on edge-case and random inputs.
• The reordered stores go to different tables, and the loop doesn't read either table between them. The shard-dirty flags are independent booleans.
• sha256 of every frame's output matches master for 8 corpora × {better, default}, on arm64 and amd64.

Benchmarks

The corpora are 32 MiB decompressed prefixes of your real corpora. Each is encoded as 4 MiB EncodeAll frames with a 4 MiB window, CRC on and concurrency 1. Each result is 10 interleaved master/PR rounds on one pinned core, compared with benchstat. Figures are SpeedBetterCompression MB/s vs master; ~ means not significant.

corpus	Graviton2 (c6g)	Graviton3 (c7g)	Graviton4 (c8g)	Sapphire Rapids (c7i)
apache.log	+12.0%	+9.9%	+8.1%	+5.1%
cockroach.node1.log	+10.0%	+4.7%	+5.0%	+3.5%
github-june-2days-2019.json	+8.6%	+7.4%	+7.8%	+3.0%
github-ranks-backup.bin	+6.4%	+4.1%	+8.0%	+2.3%
gob-stream	+8.7%	+7.3%	+10.1%	+4.2%
nyc-taxi-data-10M.csv	~	~	+5.6%	+2.5%
rawstudio-mint14.tar	+10.2%	+6.9%	+11.8%	+0.7%
sofia-air-quality-dataset.tar	+4.7%	~	+4.1%	+1.9%
geomean	+7.9%	+5.9%	+7.5%	+2.9%

All non-~ cells are p<0.05. On Zen 3 (5600X) the geomean is +4.4% with no regressions, but that box is shared, so its per-corpus spread is wide.

Follow-up

hashPrimesInRegs is named for reuse: the default-level encoder (enc_dfast.go) hashes with the same two primes, so it gets the same treatment in one commit on top of this branch, honeycombio:lizf.zstd-dfast-hash-primes: diff on top of this PR (combined diff vs master). Output is byte-identical to master. At the default level it's +4.4% on Graviton2, +2.1% on Graviton3, +1.9% on Graviton4 and flat on Sapphire Rapids (benchstat, same corpora and setup as above). I can open it as a follow-up PR once this one lands, or push it onto this PR if you'd rather review both together.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Performance
  • Improved compression hash calculations, with architecture-specific handling for hash constants.
• Tests
  • Added checks confirming the updated hash calculations match existing results across boundary values and deterministic random inputs.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Advanced

Run ID: 84799ded-b311-4fc2-8afd-bbe8fb62c068

📥 Commits

Reviewing files that changed from the base of the PR and between 87076fe and 5a2029b.

📒 Files selected for processing (2)

• zstd/encoder_test.go
• zstd/hash.go

Included review availability: Your plan provides up to 8 included reviews per hour; 6 remain after this review.

---

📝 Walkthrough

Walkthrough

The better encoder and dictionary encoder use explicit-prime hash helpers for long and short hashes. Prime selection depends on the target architecture. Tests compare the new helpers with hashLen.

Changes

Better encoder hashing

Layer / File(s)	Summary
Hash helpers and prime selection zstd/enc_better.go, zstd/hash.go, zstd/encoder_test.go	Adds long and short hash helpers that accept explicit primes. Selects primes based on the target architecture. Tests compare the helper results with hashLen for fixed and seeded random inputs.
Encoder hash integration zstd/enc_better.go	Updates both encoder paths to use the helpers for hash calculations and post-match indexing. Rewrites the indexing loops to use a computed end bound.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Refactor

Suggested reviewers: klauspost

Merge Risk: ⚪ Minimal · up to 5a202

No issue identified in the reviewed hash and encoder changes prevents merging after normal checks.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 6 functions across 8 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main changes: faster better-encoder hashing and keeping hash primes in registers on arm64.

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[klauspost]

Nice! I am focused on getting a release out, so perf PRs will have less priority for a while.

[lizthegrey]

Nice! I am focused on getting a release out, so perf PRs will have less priority for a while.

Quite alright, and yeah, that would explain why #1212 has been sitting a while, but no rush, I maintain my own branch of cherrypicked fixes for Honeycomb internal use.

[lizthegrey]

and go compiler issue opened: golang/go#81755 -- but it'll only fix it in 1.28 and newer (if it's even accepted upstream), so we need to carry the workaround.