Skip to content

Conversation

@Abdulsametileri
Copy link
Member

@Abdulsametileri Abdulsametileri commented Oct 16, 2025

related #34

Short Description (for PR summary)

Implements a snapshot mechanism to capture existing database rows before starting CDC.
Features include crash recovery, checkpoint-based resume, configurable batch processing,
and full PostgreSQL type support. Includes comprehensive example and monitoring metrics.

@Abdulsametileri Abdulsametileri changed the title feat: add snapshot feature like debezium for initial data feat: Add Debezium-style Snapshot for Initial Data Capture Oct 16, 2025
@Abdulsametileri Abdulsametileri added the enhancement New feature or request label Oct 16, 2025
@3n0ugh
Copy link
Contributor

3n0ugh commented Oct 17, 2025

All looks good for the MVP.

For the next step, we could support multiple-instance processing for the initial snapshot. We can’t apply this to normal slot capturing because of PostgreSQL limitations, but it should be feasible for the initial state.

If we run the initial snapshot with multiple instances, we need to design how they’ll be processed and ensure an instance behaves consistently when passive (i.e., a passive instance must not interfere with processing). We should also plan for failure cases — how ownership is recovered, how timeouts are handled, and how we guarantee progress if an instance goes passive or crashes.

For the next implementation, I suggest a chunk table with partial ranges at snapshot initialization, for example:

chunk_id  | start | end  | status (todo/inprogress/done) | process_start_date | created_date
uuid2343  | 0     | 99   | todo                          | NULL               | 2025-10-17T12:00
uuid2344  | 100   | 199  | todo                          | NULL               | 2025-10-17T12:00

Process the snapshot chunk-by-chunk; once all chunks covering the snapshot are finished, we can start consuming slot events.

Copy link

@emreodabas emreodabas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@Abdulsametileri Abdulsametileri merged commit 7fbb069 into main Nov 24, 2025
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants