Repository navigation
Copy overlapping matches in bulk - #143
Merged
Merged
Conversation
falklindner
changed the title
falklindner
changed the title
falklindner
added a commit
to falklindner/dissect.util
that referenced
this pull request
Oct 26, 2026
fox-it#143 adds a test at the same place in test_lzo.py. With this one after the benchmark test, the two PRs merge cleanly in either order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HJvjcGbjsG74UunzfT5zNE
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #143 +/- ##
=====================================
Coverage 0.00% 0.00%
=====================================
Files 31 31
Lines 2170 2180 +10
=====================================
- Misses 2170 2180 +10
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Schamper
requested changes
Oct 27, 2026
| @@ -40,7 +40,12 @@ def decompress(src: bytes | BinaryIO) -> bytes: | |||
| if distance > len(dst): | |||
| raise ValueError("Invalid match distance in ADC stream") | |||
Contributor
There was a problem hiding this comment.
Suggested change
| raise ValueError("Invalid match distance in ADC stream") | |
| raise ValueError("Invalid match distance") |
Might as well make this the same error message.
Contributor
Author
There was a problem hiding this comment.
Done in 1afd205.
🤖 Generated with Claude Code
Merging this PR will improve performance by 99.23%1 warning
Performance Changes
Tip Curious why performance improved? Comment Comparing 
|
Schamper
approved these changes
Oct 27, 2026
yunzheng
pushed a commit
to falklindner/dissect.util
that referenced
this pull request
Oct 27, 2026
fox-it#143 adds a test at the same place in test_lzo.py. With this one after the benchmark test, the two PRs merge cleanly in either order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HJvjcGbjsG74UunzfT5zNE
The pure Python LZO, LZXPRESS and LZXPRESS Huffman decompressors did not check that a match refers back to data that is actually there. A corrupt stream with a match before the start of the output silently produced wrong output, as slicing the output clamps out of range indices. Raise a ValueError for such a match instead, as the ADC, LZVN and LZFSE decompressors already do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LNZJRJFni88V8JGtP2tj29
The pure Python LZO, LZXPRESS, LZXPRESS Huffman, LZVN and LZFSE decompressors copied an overlapping match one distance worth of bytes at a time, and ADC copied every match one byte at a time. A long match with a short distance took many small appends to the output. An overlapping match repeats the last distance bytes of the output, so append those in bulk instead, in at most two steps per match. Matches that do not overlap are copied with a single slice, as before. This is the same change as made to LZ4 in fox-it#141. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LNZJRJFni88V8JGtP2tj29
ADC now raises ValueError("Invalid match distance"), like the other
decompressors.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HJvjcGbjsG74UunzfT5zNE
yunzheng
force-pushed
the
feature/bulk-match-copy
branch
from
October 27, 2026 17:17
1afd205 to
97bfaf4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to the suggestion in #141: the same bulk copy of overlapping matches, applied to the other pure Python decompressors.
ValueError("Invalid match distance")for a match before the start of the output, like ADC, LZVN and LZFSE. Before, they silently returned wrong data.Snappy and LZNT1 already copy in bulk. LZBITMAP decides literal or match per output byte, so it's unchanged.
Performance (CodSpeed simulation, estimated cycles,
main→ this PR):Testing: run
tox -e py3,native. New tests cover the distance check. I also fuzzed the bulk copy against the old loop: identical output or the identical error on 12,000 mutated inputs, plus 600 liblzfse round-trips.The native LZO (lzokay-native 0.1.0) panics on such a stream instead of raising an error, so the LZO test only covers the pure Python implementation.