CASSANDRA-21066 Full Repair for Tracked Keyspaces #4565
+4,631
−1,880
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
maedhroz
force-pushed
the
CASSANDRA-21066-maedhroz
branch
from
9edc6c2 to
d74f11b
Compare
maedhroz
force-pushed
the
CASSANDRA-21066-maedhroz
branch
from
9ce9758 to
673e86f
Compare
maedhroz
commented
Jan 24, 2026
maedhroz
commented
Jan 25, 2026
maedhroz
force-pushed
the
CASSANDRA-21066-maedhroz
branch
from
f876476 to
7781a20
Compare
maedhroz
commented
Feb 1, 2026
maedhroz
commented
Feb 5, 2026
|
|
||
| import org.junit.Test; | ||
|
|
||
| public class TrackedNonZeroCopyRepairTransferTest extends TrackedRepairTransferTest |
Contributor
Author
There was a problem hiding this comment.
These tests, and their ZCS equivalents could probably just share a cluster and create a keysapce per test.
…aces This comes with lots of comments and a handful of integration tests, but the tests aren't passing and it coul use more unit test coverage. There's a refactor (introducing SyncTasks) that's only partially complete. The rest of this commit message is a description of how full repair is intended to work for tracked keyspaces. Tracked keyspaces cannot accept new data without first registering it in the log. Any unreconciled data that isn't present in the log will break read monotonicity, since mutation tracking uses a single data read and can only read reconcile mutation IDs that are present in the log. For more information about how bulk transfers work on tracked keyspaces, see TrackedImportTransfer. Full repair sync tasks also deliver data to replicas, and require integration with the log just like imports do. For more details on a read anomaly that could happen without integration with the bulk transfer machinery, see TrackedKeyspaceRepairSupportTest#testFullRepairPartiallyCompleteAnomaly. The general design of this integration is to give repair SyncTasks the same two-phase commit as import transfers, where we stream SSTables to a pending/ directory, then once sufficient streams complete successfully, we "activate" those streams and move them out of the pending directory and into the live set. The first step is to ensure that each SyncTask is aligned to a single Mutation Tracking shard, by splitting SyncTasks along the shard boundaries. Each SyncTask will then stream data within a single shard, and permit us to assign a single transfer ID to each SyncTask. Each participant in a repair may receive different SyncTasks (or none at all, if they're already in-sync). This means that TransferActivation needs to be made more flexible, and support a single TransferActivation with multiple plan IDs, or no plan IDs at all. This increase in flexibility has not yet been implemented. patch by Abe Ratnofsky; reviewed by Caleb Rackliffe and ? for CASSANDRA-21066
…and AbstractCoordinatedBulkTransfer
…or are different nodes
…when nodes are on the receiving end of multiple sync tasks - minor renaming and cleanup
- stop trying to reconcile not-yet-committed non-ZCS streams
…now described in TrackedRepairTransferTest
- fix for the case where a remote sync task is split along shard boundaries
- removed the unused alignedToShardBoundaries() method in MTS
maedhroz
force-pushed
the
CASSANDRA-21066-maedhroz
branch
from
a5fe898 to
4492a7f
Compare
aratno
reviewed
Feb 7, 2026
Comment on lines
280
to
281
| Preconditions.checkState(transfer instanceof TrackedImportTransfer); | ||
| result.transfers.add(transfer.id(), ((TrackedImportTransfer) transfer).sstables); |
Contributor
There was a problem hiding this comment.
We need to support TrackedRepairTransfer inclusion here to make sure summaries include activated repairs after restart.
Contributor
Author
There was a problem hiding this comment.
I'm going to attempt to do this by having CoordinatedTransfer and ActivationRequest carry Bounds instead of a Range...
Comment on lines
+64
to
+65
| this.planId = planId; | ||
| this.transferId = transferId; |
Contributor
There was a problem hiding this comment.
Pondering whether we still need planId in SyncStat if it's here now...
Contributor
Author
There was a problem hiding this comment.
I think it probably can...
Contributor
Author
There was a problem hiding this comment.
Ah, I think it's because we key local pending transfers on plan ID.
- don't try to run tests in TrackedRepairTransferTestBase and TrackedTransferTestBase
- deduplication in Shard - supporting all CoordinatedTransfer types in loadFromJournal()
maedhroz
force-pushed
the
CASSANDRA-21066-maedhroz
branch
from
d24b770 to
143eab3
Compare
maedhroz
force-pushed
the
CASSANDRA-21066-maedhroz
branch
from
55bd0f2 to
d2b6ed6
Compare
- clean up already committed check in activate()
maedhroz
force-pushed
the
CASSANDRA-21066-maedhroz
branch
from
d2b6ed6 to
a992602
Compare
… (it's possible to have a plan ID but not stream any actual data and therefor not have a pending local transfer to activate)
… (it's possible to have a plan ID but not stream any actual data and therefor not have a pending local transfer to activate)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Tracked keyspaces cannot accept new data without first registering it in
the log. Any unreconciled data that isn't present in the log will break
read monotonicity, since mutation tracking uses a single data read and
can only read reconcile mutation IDs that are present in the log. For
more information about how bulk transfers work on tracked keyspaces, see
TrackedImportTransfer.
Full repair sync tasks also deliver data to replicas, and require
integration with the log just like imports do. For more details on a
read anomaly that could happen without integration with the bulk
transfer machinery, see
TrackedKeyspaceRepairSupportTest#testFullRepairPartiallyCompleteAnomaly.
The general design of this integration is to give repair SyncTasks the
same two-phase commit as import transfers, where we stream SSTables to a
pending/ directory, then once sufficient streams complete successfully,
we "activate" those streams and move them out of the pending directory
and into the live set.
The first step is to ensure that each SyncTask is aligned to a single
Mutation Tracking shard, by splitting SyncTasks along the shard
boundaries. Each SyncTask will then stream data within a single shard,
and permit us to assign a single transfer ID to each SyncTask.
Each participant in a repair may receive different SyncTasks (or none at
all, if they're already in-sync). This means that ActivationRequest must be flexible
enough to support transfers with multiple plan IDs or none at all.