refactor: checkpoint storage GC foundation by zhanglei1949 · Pull Request #678 · alibaba/neug

zhanglei1949 · 2026-07-02T01:48:48Z

Summary

Extract the checkpoint storage core / GC foundation from PR #621 into a focused first PR.

Replace multi-checkpoint discovery with a single durable CHECKPOINT pointer plus checkpoint-N.next staging publish/discard flow.
Add storage-level CheckpointSession for DB close/recovery checkpoints, including safe publish, reopen, rollback, and old checkpoint cleanup.
Add runtime-file materialization in CheckpointFileManager and update CSR/property graph dump paths to use the staging checkpoint APIs.
Change IDataContainer::Fork / MMapContainer::Fork to shared ownership so checkpoint GC can safely remove old checkpoint directories while readers still hold containers.
Add/update checkpoint GC, file-manager, and safety coverage.

Copilot

Pull request overview

This PR extracts a “GC-ready” checkpoint storage foundation by moving the system to a single durable CHECKPOINT pointer with a checkpoint-N.next staging workflow, plus new staging-aware checkpoint/session and file materialization APIs that enable safe cleanup of old checkpoint directories.

Changes:

Introduces CheckpointManager::StagingCheckpoint and a single-current-checkpoint layout using CHECKPOINT + checkpoint-N.next publish/discard semantics.
Adds CheckpointSession and updates DB close/recovery and loader flows to checkpoint via staging + safe publish/rollback/cleanup.
Refactors container/runtime file handling to support safe GC (shared ownership of containers; runtime-file RAII; updated CSR/property dump paths) and expands checkpoint GC/file-manager tests.

Reviewed changes

Copilot reviewed 34 out of 34 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/utils/test_table.cc	Uses `make_checkpoint()` helper to avoid direct `CreateCheckpoint()/GetCheckpoint()` pairing in module tests.
tests/unittest/utils.h	Replaces checkpoint boilerplate with `make_checkpoint()` that creates standalone test checkpoints outside the staging slot.
tests/storage/test_mutable_csr.cc	Adjusts CSR dirty-path tests to reopen into new instances before mutating/dumping.
tests/storage/test_checkpoint.cc	Reworks checkpoint tests to validate single-current-checkpoint + staging publish/discard + GC behaviors.
src/utils/io/file/file_utils.cc	Adds `fsync_directory()` helper for directory entry durability after renames/links.
src/utils/exception/exception.cc	Adds `CheckpointException(msg, file_line)` overload for richer error context.
src/transaction/update_transaction.cc	Updates TP checkpoint call to pass explicit checkpoint pointer into `Dump()`.
src/storages/loader/abstract_property_graph_loader.cc	Switches loader checkpointing to staging checkpoint + commit and cleanup on failure.
src/storages/graph/vertex_timestamp.cc	Migrates runtime object publishing to runtime-file handle API.
src/storages/graph/property_graph.cc	Extends `Dump()` signature (checkpoint + reopen + wal cleanup flag) and gates obsolete WAL deletion.
src/storages/graph/graph_snapshot_store.cc	Adds `ReplaceCurrentSnapshot()` and annotates pin/unpin APIs as `noexcept`.
src/storages/graph/graph_interface.cc	Updates AP checkpoint creation to call `Dump(graph_.checkpoint_ptr())`.
src/storages/graph/checkpoint.cc	Adds snapshot-dir fsync before writing meta during `UpdateMeta()`.
src/storages/graph/checkpoint_session.cc	New: DB-level checkpoint session providing dump/publish/reopen/rollback/cleanup flow.
src/storages/graph/checkpoint_manager.cc	Implements single `CHECKPOINT` pointer model, staging directory management, and GC of non-current checkpoints.
src/storages/graph/checkpoint_file_manager.cc	Introduces runtime-file RAII + shared container ownership and updates commit/link behavior for GC safety.
src/storages/csr/mutable_csr.cc	Updates container ownership and migrates runtime object publishing to runtime-file handle API.
src/storages/csr/immutable_csr.cc	Updates container ownership to shared pointers returned from `Checkpoint::OpenFile/CreateRuntimeContainer`.
src/storages/container/mmap_container.cc	Changes `Fork()` to return shared ownership to allow GC while readers still hold mappings.
src/main/neug_db.cc	Uses `CheckpointSession` for close/recovery checkpoints; creates initial checkpoint via staging if none exists.
include/neug/utils/serialization/in_archive.h	Marks move ctor/assignment `noexcept`.
include/neug/utils/property/column.h	Updates dump/open to use shared containers and runtime-file handle API for string columns.
include/neug/utils/io/file/file_utils.h	Declares `fsync_directory()` with directory-fsync contract comment.
include/neug/utils/exception/exception.h	Declares `CheckpointException(msg, file_line)` overload.
include/neug/storages/loader/abstract_property_graph_loader.h	Switches initialization to staging checkpoint and tracks it via `std::optional`.
include/neug/storages/graph/property_graph.h	Updates `Dump()` API to accept checkpoint + optional WAL cleanup.
include/neug/storages/graph_snapshot_store.h	Adds `ReplaceCurrentSnapshot()` API and `noexcept` annotations for snapshot pin/unpin + guard.
include/neug/storages/container/mmap_container.h	Makes `Close()` override explicit and changes `Fork()` to return `std::shared_ptr`.
include/neug/storages/container/i_container.h	Adds pure virtual `Close()` and updates `Fork()` return type to `std::shared_ptr`.
include/neug/storages/container/file_mmap_container.h	Updates `Dump()` contract comment to “consuming” semantics aligned with new commit flow.
include/neug/storages/checkpoint.h	Updates checkpoint file API to shared containers and runtime-file handle commit API.
include/neug/storages/checkpoint_session.h	New: `CheckpointSession` public API.
include/neug/storages/checkpoint_manager.h	Redesigns manager to single-current checkpoint with `StagingCheckpoint` handle semantics.
include/neug/storages/checkpoint_file_manager.h	Updates file manager API to shared containers and runtime-file handle lifecycle + snapshot-dir fsync.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

refine implementation for Checkpoint session and checkpoint manager fix ckp.commit(), make hardlink runtime file to next ckp snapshot format refine implementation refine refine

liulx20 · 2026-07-15T08:10:10Z

  void openGraphAndIngestWals();
  void ingestWals(IWalParser& parser, PropertyGraph& graph);
  void initPlannerAndQueryProcessor();
+  std::shared_ptr<Checkpoint> consumeLiveGraphAndCommitCheckpoint(


Why does this interface require the MemoryLevel parameter?

longbinlai

LGTM

zhanglei1949 marked this pull request as ready for review July 2, 2026 02:58

zhanglei1949 force-pushed the codex/checkpoint-storage-gc-foundation branch from be94cd8 to eee4447 Compare July 14, 2026 02:36

zhanglei1949 requested a review from Copilot July 14, 2026 02:38

Copilot started reviewing on behalf of zhanglei1949 July 14, 2026 02:39 View session

Copilot AI reviewed Jul 14, 2026

View reviewed changes

Comment thread src/storages/graph/checkpoint.cc

Comment thread src/utils/io/file/file_utils.cc

Comment thread src/storages/graph/graph_snapshot_store.cc Outdated

zhanglei1949 commented Jul 14, 2026

View reviewed changes

Comment thread src/main/neug_db.cc Outdated

zhanglei1949 added this to neug Jul 14, 2026

github-project-automation Bot moved this to In progress in neug Jul 14, 2026

zhanglei1949 requested a review from luoxiaojian July 15, 2026 02:37

refactor: add checkpoint storage GC foundation

0c03b73

refine implementation for Checkpoint session and checkpoint manager fix ckp.commit(), make hardlink runtime file to next ckp snapshot format refine implementation refine refine

zhanglei1949 force-pushed the codex/checkpoint-storage-gc-foundation branch from eaa4730 to 0c03b73 Compare July 15, 2026 04:26

zhanglei1949 added 2 commits July 15, 2026 13:00

minor fix

709bf44

fix test

dd993de

zhanglei1949 requested a review from liulx20 July 15, 2026 06:10

zhanglei1949 added 2 commits July 15, 2026 14:34

minor

68ffb4e

fix test

9505487

zhanglei1949 requested a review from longbinlai July 15, 2026 07:10

liulx20 reviewed Jul 15, 2026

View reviewed changes

Comment thread include/neug/main/neug_db.h

zhanglei1949 added 3 commits July 15, 2026 16:28

introduce error_code ERR_NO_CHECKPOINT

21078ac

fix test

dd4ce90

Merge branch 'main' into codex/checkpoint-storage-gc-foundation

12285f3

longbinlai approved these changes Jul 15, 2026

View reviewed changes

zhanglei1949 merged commit 7d7483d into alibaba:main Jul 16, 2026
16 checks passed

github-project-automation Bot moved this from In progress to Done in neug Jul 16, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

refactor: checkpoint storage GC foundation - #678

refactor: checkpoint storage GC foundation#678
zhanglei1949 merged 8 commits into
alibaba:mainfrom
zhanglei1949:codex/checkpoint-storage-gc-foundation

zhanglei1949 commented Jul 2, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

liulx20 Jul 15, 2026 •

edited

Loading

Uh oh!

zhanglei1949 Jul 15, 2026

Uh oh!

Uh oh!

longbinlai left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

zhanglei1949 commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

liulx20 Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhanglei1949 Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

longbinlai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zhanglei1949 commented Jul 2, 2026 •

edited

Loading

liulx20 Jul 15, 2026 •

edited

Loading