Conversation
arminsabouri
force-pushed
the
block-indexer
branch
from
394ad06 to
f71c3fc
Compare
arminsabouri
force-pushed
the
tip-to-depth
branch
from
17a35fc to
3ff46de
Compare
arminsabouri
force-pushed
the
block-indexer
branch
2 times, most recently
from
b8c0e1c to
bc9d6c7
Compare
arminsabouri
force-pushed
the
tip-to-depth
branch
from
3ff46de to
f30ae93
Compare
arminsabouri
force-pushed
the
tip-to-depth
branch
from
f30ae93 to
ae4123c
Compare
arminsabouri
force-pushed
the
tip-to-depth
branch
3 times, most recently
from
f7af22b to
9bcd4be
Compare
arminsabouri
force-pushed
the
tip-to-depth
branch
3 times, most recently
from
837d9b5 to
fb32de4
Compare
| @@ -0,0 +1,4 @@ | |||
| Recovered manifest with next_file=5 manifest_num=4 log_num=3 prev_log_num=0 last_seq=0 | |||
There was a problem hiding this comment.
Do we need these files? I dont think so. only .log blobs. Undo files may also be not needed
There was a problem hiding this comment.
since tx-indexer indexes txs but does not maintain the state of the UTXO set, I don't think it needs rev*dat (undo/reorg). LOG and LOG.old seems not needed either
|
Need more review before merging. |
im doing it rn |
| /// every blk file with it (cycling the key by absolute byte position). This struct reads | ||
| /// the key once on construction and applies it transparently in all read operations. | ||
| /// When `xor.dat` is absent or all-zero the reads are pass-through. | ||
| pub struct BlkFileStore { |
There was a problem hiding this comment.
is it worth to align with lib.rs for consistency? pub(crate) there and pub here
| @@ -0,0 +1,4 @@ | |||
| Recovered manifest with next_file=5 manifest_num=4 log_num=3 prev_log_num=0 last_seq=0 | |||
There was a problem hiding this comment.
since tx-indexer indexes txs but does not maintain the state of the UTXO set, I don't think it needs rev*dat (undo/reorg). LOG and LOG.old seems not needed either
| let obfuscation_key = db | ||
| .get(OBFUSCATE_KEY_KEY) | ||
| .map(|b| b.to_vec()) | ||
| .map(|b| b[1..].to_vec()) |
There was a problem hiding this comment.
what if b is empty? would It be better to add guards?
There was a problem hiding this comment.
If the result is empty there is not much we can do. Panic here seems appropriate bc it suggests level DB is serving corrupt keys (?)
| .expect("failed to open sled DB") | ||
| .spk_db() | ||
| .expect("failed to open spk_db tree"); | ||
| // 1. Build indices into a temp directory (or user-specified index-dir) |
There was a problem hiding this comment.
nit: update the comment
maybe just Build indices into a temp directory is enough
| } | ||
| file_hints.push((file_no, info.height_first, info.height_last)); | ||
| } | ||
| Err(_) => continue, |
There was a problem hiding this comment.
can silent skip produce an incomplete index?
There was a problem hiding this comment.
Yup great catch
|
|
||
| let spec = DenseBuildSpec { | ||
| blocks_dir, | ||
| range: start_height..end_height + 1, |
There was a problem hiding this comment.
Still new to codebase so just asking
Should this dense range start at start_height without adjusting the block height accessors too? Once the storage is offset from genesis block_of_tx() returns global heights but tx_range_for_block() still indexes block_tx_index as if the first stored block were height 0 so tx_range_for_block(block_of_tx(...)) will panic for tip windows.
There was a problem hiding this comment.
yes your intuition is correct
And getting block info from some block to some depth by iterating the chain backwards
Add multi-file parser support and sync_from_tip entry point that uses BlockIndex to find the tip, walks back depth blocks, and indexes forward into dense storage.
arminsabouri
force-pushed
the
tip-to-depth
branch
from
fb32de4 to
bd29ca9
Compare
We should passing input datadir, where we want the data to go and depth. Everything else can be abstract away in the internal methods
If requested depth exceeds chain length return a specific error instead of key not found.
Ensure we can parse multiple blk files.
arminsabouri
force-pushed
the
tip-to-depth
branch
from
bd29ca9 to
13f195e
Compare
arminsabouri
force-pushed
the
tip-to-depth
branch
from
13f195e to
67bb4c2
Compare
3 participants
This Pr just focuses on index building from the tip to some desired depth by first identifying the best block and working backwards to the depth and parsing from there. Everything in the dense id system work as is, treating the nth block at if it was genesis. i.e everything is stil the ith order just relative to the depth at which we sync at.