Scanner engine: dedup + candidate-filter perf - #5
Merged
Conversation
Scans the PE64/ELF64 fixtures via matches_code with patterns whose strongest literal run is not the leading one (leading-wildcard / late-rare), plus a strong-leading control. Used to track candidate-filter improvements.
finds_unique_direct rebuilt the 256-byte Boyer-Moore skip table on every span. Thread the precomputed PreparedPattern/PatternPlan anchor_jumps through the direct quick-search path instead, matching what the Matches path already does.
Matches::next routed through its own copies of the linear / first-byte / quick search strategies, duplicating ~250 lines against the finds path. Route it through the shared find_next_in_span instead and delete the Matches copies; drop the now-redundant _direct suffix since these are the sole implementation. The matches/finds cross-check proptests confirm behavior is unchanged.
Every candidate verification built a fresh ExecReader whose constructor ran a find_span binary search to locate the span the candidate was already inside. Record the current span index in ExecScratch (set per span in find_next_in_span) and seed ExecReader::new with it; find_span checks the seed first, so in-span reads skip the binary search. Out-of-span reads (after a jump) self-correct.
exec_backtracking reconstructed an ExecReader (running find_span) for every popped state. Build it once before the state loop and reuse it; the span cache persists across states and self-corrects on jumps.
finds_unique allocated a fresh executor scratch and a second-match probe vec on every call. Introduce a public FindScratch (executor scratch + probe buffer) and finds_prepared_with so hot loops over many patterns can reuse it across calls; finds_code/finds_prepared keep their simple signatures via a local FindScratch.
Anchor selection only scanned the leading byte run, so patterns with an early wildcard (e.g. "? ? 48 8b 0d ...") got a weak anchor or none and degraded to a near-linear scan. Build an offset map of known bytes across wildcards/skips/reads and pick the most selective fixed-byte window anywhere in it. Pure filter; the full pattern still verifies each candidate. Replaces build_prefix/select_anchor.
Add coverage for the offset-map anchor beyond the leading run: a skip that pushes the distinctive run deep into the prefix (mapped + selected as a deep anchor), the ANCHOR_MAP_CAP boundary (a run past the cap is dropped), and a scan that still matches when the strong run is beyond the cap (weak anchor + exec).
killerra
force-pushed
the
scanner-improvements
branch
2 times, most recently
from
June 26, 2026 00:23
da00958 to
76e68c5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scanner engine fixes from the code review, each as its own commit. All changes preserve
matching semantics — the anchor and span-hint are pure candidate-filter optimizations, and
the matches/finds cross-check proptests + fixture tests guard correctness.
Commits
the 256-byte table per span.
Matchesvs*_direct,~250 lines); both paths now share one linear / first-byte / quick implementation.
ExecReaderwith the scanning span (viaExecScratch) so per-candidateverification skips the
find_spanbinary search; out-of-span reads self-correct.ExecReaderacross backtrack states instead of reconstructing it(and re-
find_span-ing) per popped state.FindScratch+finds_prepared_withso repeated uniquenessscans don't reallocate executor scratch + probe buffer each call.
prefix (across wildcards/skips/reads), not just the leading run.
scan_anchorbenchmark for weak-leading / leading-wildcard patterns.Benchmark (matches_code, median;
cargo bench --bench scan_anchor)48 8b ? ? ? ? 48 89 ? ? ba 2c(weak lead)? ? 48 8b 0d ? ? ? ? 15 7c(leading wildcards)48 8b ? ? ? ? ? ? 0f b6 84(late rare run)55 41 57 41 56(strong-lead control)The big wins are the anchor (SCAN-8). The ~6% on the strong-lead control — where the anchor
is unchanged — comes from the exec-overhead fixes (SCAN-4/VM-2 removing the per-candidate
find_span).Verification
cargo test --workspace --all-targets --all-features,cargo test --workspace --doc,cargo clippy --workspace --all-targets --all-features -- -D warnings -D clippy::all, andcargo fmt --all -- --checkall pass.Not included
Remaining scanner findings (SCAN-5/6/7/9, VM-1/3/4) and the non-scanner PR (#4) are separate.