Sync performance fixes #798
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sword-Smith
requested changes
Dec 9, 2025
Member
Sword-Smith
left a comment
Sword-Smith
left a comment
There was a problem hiding this comment.
The changes to wallet_state.rs module are obviated by #805. But the changes to global state (neptune-core/src/state/mod.rs) look like a very fine idea to me. Do you want to resubmit PR with only the changes to neptune-core/src/state/mod.rs?
aszepieniec
force-pushed
the
sync-performance-fixes
branch
2 times, most recently
from
f82252c to
025b8e6
Compare
Contributor
Author
|
I had some doubts at first but after splitting the changes into incremental commits I convinced myself that it's all good. I just force-pushed essentially the same changes, spread out over more commits, and without the changes to |
Contributor
Author
|
It looks like the early returns also live in wallet_state.rs. Would you like me to remove those as well? (I read your request as pertaining to the parallelization of index derivation specifically.) |
Contributor
Author
|
Ball is in your park @Sword-Smith. The question is about the early returns in |
aszepieniec
force-pushed
the
sync-performance-fixes
branch
from
3816950 to
222957e
Compare
Caches grow quite large during restoration of 1000s of UTXOs, leading to out-of-memory errors. Credit to AllFather team.
…ived In other words, do not call the restoration method whenever the vector of UTXOs received in this block is empty. Credit to AllFather team.
Credit to AllFather team.
Iterate only over new UTXOs, not all UTXOs, in the course of MUTXO MSMP restoration. Makes syncing faster. But do this only when a flag is set, so that we can set it while syncing (and default to the full range in all other cases). Credit to AllFather team. Docstring modified by Alan.
Modify the optimization of function `restore_monitored_utxos_from_archival_mutator_set` whereby only the trailing end of the list of monitored UTXOs is processed if `process_all` is false and if `new_utxos` contains some non-empty vector. As a result of @Sword-Smith's new database schema, all necessary data for fetching a mutator set membership proof for a given UTXO lives on the `MonitoredUtxo` struct. Consequently, the corresponding data for monitored UTXOs that do not have a membership proof yet is already available -- and there is no need to pass this data as an argument. Indeed, the PR that applied that database schema change already dropped the argument `new_utxos`. Meanwhile, PR `sync-performance-fixes` used argument `new_utxos` for an optimization, namely to skip the updates of all monitored UTXOs that already have some MSMP, even if it is outdated. The relevant information contained by the argument `new_utxos` is its length (if `Some`), because based on the length of the tail we know where to set the starting index for the loop. This PR merges arguments `process_all` and `new_utxos` into one argument, `ignore_all_but_trailing` of type `Option<usize>`. Set it to `None` for the default behavior, in which case none of the monitored UTXOs will be ignored. During syncing, or other cases where we don't care about outdated membership proofs and only care about populating the `k` trailing new monitored UTXOs with some membership proofs, set it to `Some(k)`.
aszepieniec
force-pushed
the
sync-performance-fixes
branch
from
222957e to
0f19b4d
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There might be something useful in these changes. Credit to AllFather team.