Improve memory usage and speed. #223
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
To work around the zone tree storing RRSIGs as an RRSET with a single TTL which is incorrect for RRSIGs. Note: Switches from Zone to LightWeightZone so that the zone walk behaviour can be overridden, which means we lose the more proper DNS response generation, but gain reduced memory consumption.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I made these changes while working on #174 because the fix I made for that issue, #214, causes much slower signing as it seems it is quite slow to insert into the
LightWeightZone(well, that may be the cause, I'm not sure).I realized that a lot if not most of the records being inserted are duplicates of the unsigned zone data, and we can instead just refer to the original copy of the unsigned zone data. This PR changes
LightWeightZoneto take an optional additional zone which is also walked byfn walk(). The zone signer then creates the new signed zone with a reference to the underlying unsigned zone. However, one issue with this is that there is no guarantee that the unsigned data in the original zone was the same data that was used during signing, as it is not versioned nor referenced by serial number.I'm leaving this PR here as a draft, I suspect if we come back to it it will be in a different form, but we can leave this here as for discussion for now. Note that the PR isn't fully cleaned up, I want to commit it before I go on vacation so am pushing this now.
Also, at the time I hadn't yet discovered #222 which negatively impacts speed and memory by causing walking of and collecting glue records multiple times, sometimes many many times, which all get copied into the to-be-signed records collection which uses more memory and takes more time, both to collect and to handle during signing.
The impact is quite dramatic. For an old copy of the
nlzone memory usage drops from around 40-50 GiB down to 30 GiB. It also reduced total signing time quite a bit as it doesn't need to copy the unsigned records to the signed zone. It is however a bit messy in the signing code to use a combination of unsigned records plus generated records as input to RRSIG generation and then include the generated unsigned records along with the signed records in the final zone, while leaving the original unsigned records only linked to the zone via a reference to the real unsigned zone.This PR also includes #219.
Some quick screenshots I made while testing are attached blow: