feat(chain): add snapshot loading support to database and chain plugin#87
Open
feat(chain): add snapshot loading support to database and chain plugin#87
Conversation
Member
On1x
commented
Apr 12, 2026
On1x
commented
- Implement database::open_from_snapshot to initialize DB from snapshot state
- Add database::initialize_hardforks to setup hardfork data post-snapshot
- Extend database interface with snapshot-related methods and fork database accessors
- Reflect new objects: content_type_object, content_vote_object, witness_vote_object
- Modify chain plugin to accept --snapshot option and open DB in snapshot mode
- Integrate snapshot plugin into vizd application and update CMakeLists accordingly
- Include snapshot plugin header and register it with appbase plugin system
- Update submodules chainbase and fc to newer commits
- Implement database::open_from_snapshot to initialize DB from snapshot state - Add database::initialize_hardforks to setup hardfork data post-snapshot - Extend database interface with snapshot-related methods and fork database accessors - Reflect new objects: content_type_object, content_vote_object, witness_vote_object - Modify chain plugin to accept --snapshot option and open DB in snapshot mode - Integrate snapshot plugin into vizd application and update CMakeLists accordingly - Include snapshot plugin header and register it with appbase plugin system - Update submodules chainbase and fc to newer commits
- Add flush() call after writing snapshot data to file to ensure buffered data is written - On non-Windows systems, open the file and call fdatasync() to persist data to disk - Enhance crash safety by guaranteeing snapshot file integrity after write operation
…lugin - Provide standard Docker run command with port mappings and volume mounts - Explain volume mappings for config files and blockchain data - Describe use of VIZD_EXTRA_OPTS for passing extra CLI arguments - Detail how to create periodic snapshots via config and container restart - Include instructions for one-shot snapshot creation with container run - Outline steps to create snapshot at specific block number - Explain loading snapshot during new node bootstrap with environment variables - Emphasize restarting container without extra opts after initial snapshot load - Suggest host-side cron job for managing snapshot retention - Add quick reference table for common snapshot-related Docker commands
- Updated fc subproject commit from 628882d to be9d885 - Ensured thirdparty/fc points to latest commit for sync
…plugin - Added FC_REFLECT for required_approval_object with id, account, proposal fields - Removed unused type alias in import_objects_generic function - Implemented custom serialization of proposal_index objects to handle shared allocators - Converted bip::flat_set fields to fc::flat_set in proposal_object export - Serialized buffer_type field to std::vector<char> for JSON compatibility - Preserved all approval-related sets and public key approvals in exported state - Replaced default EXPORT_INDEX macro usage with manual export for proposal_index
- Add conditional assertion checking vesting withdraw rate only after hardfork 4 - Prevent unnecessary assertion failure before the specified hardfork - Ensure compatibility with chain rules for vesting withdrawals
… plugin - Add snapshot_load_callback and snapshot_create_callback to chain plugin - Load snapshot state during chain plugin startup before on_sync event - Create snapshot after full DB load but before on_sync during startup - Introduce _dlt_mode flag in database to support snapshot-only mode without block log writes - Skip block log validation and fork DB seeding in DLT mode; P2P seeds fork DB after restart - Modify database code to conditionally bypass block log append and ID updates in DLT mode - Remove vesting withdraw rate hardfork conditional due to logic simplification - Register snapshot plugin callbacks in plugin_initialize for snapshot loading and creation - Improve docs to clarify snapshot creation and loading processes occur inside chain plugin startup - Ensure P2P and witness plugins never start until snapshot loading or creation completes - Add missing FC_REFLECT annotations for proposal_object related types to support snapshot serialization
…ng block_log - Add snapshot plugin documentation explaining usage, config, and protocol - Support snapshot creation, loading, and periodic snapshots in nodes - Implement TCP snapshot serving with trust model and anti-spam protection - Add snapshot P2P client to download and load snapshots from trusted peers - Introduce DLT mode detection and enable rolling block_log with configurable window - Implement block_log truncation to maintain rolling window in DLT mode - Enhance chain plugin to handle snapshot load and P2P sync callbacks safely - Add snapshot network message types and serialization helpers - Provide snapshot server logic handling connections, requests, and data streaming - Add client logic to query peers, select best snapshot, and download in chunks - Ensure clean state by wiping shared memory before importing snapshots - Clear genesis objects before snapshot import to avoid conflicts - Add CLI and config options for periodic snapshot and DLT rolling block_log size - Improve error handling on snapshot load failures with logging and app exit
- Add dlt_block_log as a separate append-only block store for recent blocks - Design offset-aware index file with 8-byte header for arbitrary start block number - Modify database to open, read, write, and truncate dlt_block_log alongside block_log - Update irreversible block handling to write to dlt_block_log in DLT mode - Implement automatic fallback to dlt_block_log for block retrieval not in main block_log - Update config and documentation to describe dlt_block_log behavior and settings - Remove old block_log truncation logic, replacing it with dlt_block_log truncation - Add new dlt_block_log.cpp and header with thread-safe memory-mapped file access - Enhance P2P layer fallback to serve blocks from dlt_block_log if missing from main log
- Changed dlt_block_log_impl to detail::dlt_block_log_impl in dlt_block_log.cpp - Ensures correct scoping of the implementation class - Improves code clarity and namespace management
- Skip past already-irreversible blocks when dlt_block_log is empty - Trust the snapshot by advancing dlt_head_num to last irreversible block - Applied this fix in multiple locations to ensure consistency - Prevent potential reprocessing of irreversible blocks after snapshot load
- Introduce snapshot_max_age_days config to specify max snapshot age - Implement cleanup_old_snapshots function to remove snapshots older than set days - Call cleanup_old_snapshots after creating periodic or block-based snapshots - Log removal actions and errors during cleanup process - Add snapshot-max-age-days option with default value 90 days - Enable snapshot rotation logging during plugin initialization when configured
- Added timing start marker before serving data requests - Accumulated bytes sent during data request handling loop - Calculated elapsed time after serving completes - Logged total bytes served and elapsed time using ilog - Updated thirdparty fc submodule commit reference
- Changed subproject commit from 1c1165a to 728d23aa - Updated thirdparty/fc to point to latest commit
…ndling - Add snapshot caching to avoid reading file on each info request - Implement async connection handling with timeout in separate fibers - Limit max concurrent connections to prevent overload (max 5) - Add atomic counter to track active connections safely - Enforce max snapshot download size of 2 GB for safety - Stream checksum calculations in 1 MB chunks to reduce memory use - Add 64 KB max payload limit for control messages (info requests) - Implement rate limiting and cleanup for connection history map - Reject duplicate sessions from the same IP concurrently - Switch snapshot serving to use cached snapshot info and file path - Stream file chunks during snapshot data serving to minimize memory - Improve snapshot download verification by streaming checksum check - Handle socket closing and error logging properly on connection end
…ails - Enhance `--snapshot` option description with restart safety and file rename after import - Add new CLI and config options for snapshot sync from trusted peers and snapshot serving over TCP - Document built-in snapshot rotation and disk space management configurable by `snapshot-max-age-days` - Introduce detailed explanation on DLT mode, rolling block log, and P2P snapshot sync mechanism - Provide restart safety scenarios and clarify snapshot import skip conditions - Update recommended production config with new snapshot and DLT block log settings - Add usage examples for P2P auto-bootstrap and force re-import scenarios - Expand component changes overview reflecting core system modifications for snapshot and DLT block log - Improve connection handling logic in snapshot server plugin for synchronous fiber-based operation - Refactor chain plugin startup to skip snapshot import if shared memory exists or snapshot file missing - Rename snapshot file to `.used` post-import to prevent duplicate imports on restart
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.