Feat/overlay support#94

Merged

dryajov merged 184 commits intomainfrom

feat/overlay-support

Mar 16, 2026

Commits on Mar 13, 2026

wip
dryajov
committed
fix(nimble): use submodules instead of nimble dependencies

markspanbroek
authored and
dryajov
committed
wip: reworking with kvstore and overlays
dryajov
committed
feat: fixing bugs and minor cleanup
dryajov
committed
wip
dryajov
committed
feat: add overlay operations
dryajov
committed
rename keys
dryajov
committed
feat: add putLeafsAndBlocks support
dryajov
committed
misc
dryajov
committed
add hash support for block types
dryajov
committed
wip tests
dryajov
committed
use cid for hashing
dryajov
committed
feat: handle bitseqs and duplicate blocks with different leafs
dryajov
committed
mic cleanup
dryajov
committed
feat: expose new putLeafAndBlock methods on network store, fix minor issues with dispatching
dryajov
committed
feat: migrate to kvstore, drop leveldb and nim-datastore
dryajov
committed
chore: remove TempLevelDb and CacheStore from tests, use in-memory SQLiteKVStore
dryajov
committed
fix: missing poseidon2digest import in builder, restore marketplace test examples
dryajov
committed
feat: overlay tests
dryajov
committed
rebasing main
dryajov
committed
bump submodules
dryajov
committed
ignore files
dryajov
committed
feat: remove datastore deps
dryajov
committed
refactor: migrate chunker to result-based api and update callers
dryajov
committed
feat: add bitseq combineSafe utility
dryajov
committed
refactor: simplify treehelper error handling
dryajov
committed
refactor: improve overlay and repostore operations
dryajov
committed
feat: migrate node to overlay semantics
dryajov
committed
feat: migrate erasure to overlay semantics
dryajov
committed
feat: migrate slot builder to overlay semantics
dryajov
committed
refactor: update test helpers for overlay migration
dryajov
committed
refactor: update tests for overlay migration
dryajov
committed
fix: update call sites for dual-store overlay semantics
dryajov
committed
refactor: remove manifest param and ensureExpiry, simplify overlay signatures
dryajov
committed
fix: handle uninitialized bitseq in combineSafe and overlay creation
dryajov
committed
chore: checkpoint overlay and erasure migration
dryajov
committed
feat: implement withTmpOverlay dataset store
dryajov
committed
misc: proper casts
dryajov
committed
fix: put proofs under the correct tree on upload
dryajov
committed
misc: added todo in engine
dryajov
committed
misc: formatting
dryajov
committed
fix: tidy up overlay decoding lifecycle
dryajov
committed
feat: proper upsert for availability
dryajov
committed
fix: shutdown sequence
dryajov
committed
fix: avoid sigsegv on shutdowns
dryajov
committed
fix: don't crash on nil proof
dryajov
committed
feat: do proper leaf merge
dryajov
committed
fix: proper nil type result handling
dryajov
committed
feat: handle cancellations better in overlays
dryajov
committed
fix purchasing tests
dryajov
committed
feat: updated expiry handling
dryajov
committed
fix: re-enable and fix failing tests
dryajov
committed
format
dryajov
committed
debug: allow all jobs to complete
dryajov
committed
feat: handle deletes with overlays (first pass)
dryajov
committed
feat: remove old expiry handling from node
dryajov
committed
feat: return proper error on missing dataset delete
dryajov
committed
re-enable import (do we need it?)
dryajov
committed
feat: use putLeafAndBlock in engine
dryajov
committed
feat: remplement maintenance and avoid concurrent deletes
dryajov
committed
fix: test maintenance with overlays
dryajov
committed
feat: make sure download overlay is tracked
dryajov
committed
feat: remove expiry from download flow
dryajov
committed
feat: rework repostore type serde with pb
dryajov
committed
feat: use protobuf for speed in codecs
dryajov
committed
feat: store manifest cid in overlay for easy manifest retrieval
dryajov
committed
fix: avoid sigsegv during shutdown
dryajov
committed
fix: close stream on comple in node.store
dryajov
committed
feat: re-enable common tests
dryajov
committed
feat: initial blocks bitmap optimizations
dryajov
committed
feat: cleanup block bitmap handling and more tests
dryajov
committed
misc: get rid of asyncchecksuit
dryajov
committed
feat: properly merge overlay data on updates
dryajov
committed
fix: remove unused import
dryajov
committed
bump deps
dryajov
committed
feat: consolidate overlay case putorupdate
dryajov
committed
format
dryajov
committed
format
dryajov
committed
bump kvstore
dryajov
committed
Fix nimgroth16 zero-proof bug: assert -> doAssert in converters.nim
dryajov
committed
bump metrics and kvstore
dryajov
committed
bump kvstore after format
dryajov
committed
feat: use batched versions to put proofs
dryajov
committed
misc: formatting
dryajov
committed
feat: make finilize safe
dryajov
committed
feat: remove size from block meta
dryajov
committed
feat: add downloading type to distinguish from storing
dryajov
committed
feat: remove size from block meta
dryajov
committed
feat: rename update to sync and syncUnsafe
dryajov
committed
feat: use syncModules & syncUnsafe - to avoid clashing with nimble builtins
dryajov
committed
misc: format
dryajov
committed
misc: bump kvstore
dryajov
committed
misc: move sync tasks to the bottom
dryajov
committed
Add E2E testing infrastructure
dryajov
committed
Update E2E infrastructure: fix test script, prometheus config, add dashboard
dryajov
committed
Add kvstore metrics and update E2E monitoring config
dryajov
committed
Update Grafana dashboard with comprehensive kvstore metrics
dryajov
committed
replace oid with rng
dryajov
committed
feat: properly handle re-upload with same treeCid (same data) in finilize
dryajov
committed
feat: properly track slots with overlays
dryajov
committed
misc: update grafana dashboard
dryajov
committed
bump kvstore
dryajov
committed
feat: standardize manifest tracking and dataset deletion
dryajov
committed
feat: delete dataset safely during cleanup for temp overlays
dryajov
committed
misc: cleanup tests
dryajov
committed
feat: adding grafana, prometheus and updating metrics
dryajov
committed
misc: drop unneeded test files
dryajov
committed
feat: dispose listBlocks iterator in discovery
dryajov
committed
fix: drop overlay with refcount reconsiliation
dryajov
committed
bump kvstore
dryajov
committed
bump kvstore
dryajov
committed
feat: added batching for downloads
dryajov
committed
feat: remove blockaddress type - constrain to blockexchange engine only
dryajov
committed
feat: speed up download by reading more bytes from storestream
dryajov
committed
feat: avoid excessive string allocation
dryajov
committed
feat: add batched get block
dryajov
committed
feat: batched getblocks
dryajov
committed
remove blockaddress methods from the store
dryajov
committed
feat: avoid excessive string & key reallocations
dryajov
committed
feat: batch block reads - use getBlocks
dryajov
committed
feat: use batched downloads with getBlocks
dryajov
committed
misc: small cleanup
dryajov
committed
perf: replace per-block hasBlock with single bitmap fetch in fetchBatched
dryajov
committed
feat: fix storestream hanging on no blocks
dryajov
committed
bump nim-kvstore
dryajov
committed
feat: add fsds options
dryajov
committed
feat: add overlay caching
dryajov
committed
wip
dryajov
committed
feat: passthrough fsstore args
dryajov
committed
feat: ignore default circuit locations during tests
dryajov
committed
bump kvstore
dryajov
committed
fix: make sure cache and middleware updates are consistent
dryajov
committed
fix: don't override block bits in middleware
dryajov
committed
fix: overlay cache eviction order
dryajov
committed
feat: add upload batching
dryajov
committed
misc: debug helper comment
dryajov
committed
bump kvstore
dryajov
committed
bump kvstore
dryajov
committed
bump kvstore
dryajov
committed
feat: cleanup and consolidation of blockstore public api with all calls having a batched counterpart
dryajov
committed
feat: add tests for batched operations
dryajov
committed
feat: use new blockstore api
dryajov
committed
feat: skip non-manifest blocks
dryajov
committed
feat: adding metrics
dryajov
committed
feat: add overlay helpers to tests
dryajov
committed
feat: run install.nims before test tasks to build deps
dryajov
committed
bump kvstore
dryajov
committed
fix tests
dryajov
committed
feat: expose getBlocksAndProof on netowork engine and minor cleanup
dryajov
committed
format
dryajov
committed
bump kvstore
dryajov
committed
Update archivist/blockexchange/engine/advertiser.nim

dryajov
and
markspanbroek
committed
Update archivist/blockexchange/engine/advertiser.nim

dryajov
and
markspanbroek
committed
Update archivist/blockexchange/engine/advertiser.nim

dryajov
and
markspanbroek
committed
Update tests/archivist/marketplace/sales/mockstorage.nim

dryajov
and
markspanbroek
committed
Update archivist/blockexchange/engine/engine.nim

dryajov
and
markspanbroek
committed
feat: fixup grafana dashboard
dryajov
committed
fix: fixup metrics
dryajov
committed
addressing review comments
dryajov
committed
addressing review comments
dryajov
committed
remove log statments
dryajov
committed
feat: move verifyBlockBitState to test helpers
dryajov
committed
feat: use checkBitmask more consistently
dryajov
committed
feat: move to test helpers
dryajov
committed
formatting
dryajov
committed
feat: batch proofItems
dryajov
committed
feat: add catchAsync to correctly propagate cancellations
dryajov
committed
bump kvstore
dryajov
committed
feat: don't doublecount batches
dryajov
committed
fix: store the recovered data under the correct treeCid
dryajov
committed
fix: don't decode invalid status enum vals
dryajov
committed
fix: don't decode invalid Naturals & enum vals
dryajov
committed
fix: invalidate cache and use catchAsync
dryajov
committed
feat: delete only when refCount == 0
dryajov
committed
feat: don't allow putting blocks for an overlay that is deleted
dryajov
committed
feat: refactor repostore tests
dryajov
committed
fix: delete manifest blocks from repoDs directly.
dryajov
committed
bump archivist dht
dryajov
committed
fix: circuitDir now properly uses the runtime datDir
dryajov
committed
merigin main
dryajov
committed
bumping archivistdht
dryajov
committed

Commits on Mar 16, 2026

rename block cli options to overlay
dryajov
committed
feat: add 404 param when cid not found locally
dryajov
committed
fix: use overlay terminology instead of block
dryajov
committed