CASSANDRA-17258: Add write threshold warnings for large partitions and tombstones #4556

minal-kyada · 2026-01-12T07:41:43Z

Summary

Implements replica-side write threshold warnings to detect and warn about writes to large partitions or partitions with many tombstones. This follows the same architecture as the existing read threshold warnings but is warning-only (no blocking behavior).

Related Work

This implementation follows the pattern established by read threshold warnings but adapts it for write operations with a warning-only approach.

patch by Minal Kyada reviewed by TBD for CASSANDRA-17258

dcapwell · 2026-01-12T19:40:24Z

src/java/org/apache/cassandra/config/Config.java


+    public volatile boolean write_thresholds_enabled = false;
+    public volatile DataStorageSpec.LongBytesBound write_size_warn_threshold = null;
+    public volatile int write_tombstone_warn_threshold = 1000;


use -1 for disabled. The reason that

public volatile int tombstone_warn_threshold = 1000; public volatile int tombstone_failure_threshold = 100000;

have values is that we retro fit those old behaviors into read_thresholds, which fixed their behavior to work better. They are the only configs enabled by default and have been enabled for years.

read thresholds has been around for awhile, so I might send a patch to enable things by default, but that still won't change this patch as this patch adds a new feature; so should be off by default.

dcapwell · 2026-01-12T19:44:53Z

src/java/org/apache/cassandra/cql3/statements/ModificationStatement.java

+        {
+            for (PartitionUpdate update : mutation.getPartitionUpdates())
+            {
+                totalTombstones += update.affectedRowCount();


this method doesn't have anything to do with tombstones

dcapwell · 2026-01-12T19:46:43Z

src/java/org/apache/cassandra/cql3/statements/ModificationStatement.java

+        {
+            for (PartitionUpdate update : mutation.getPartitionUpdates())
+            {
+                totalSize += update.dataSize();


is this the metric we actually want? this is how large a single mutation is, but we won't warn that its writing to a partition thats "too large"?

dcapwell · 2026-01-12T19:47:30Z

src/java/org/apache/cassandra/cql3/statements/ModificationStatement.java

+            maybeWarnWriteSize(mutations, options);
+            maybeWarnWriteTombstones(mutations, options);


why is this logic here rather than in the write coordination like read is? Wasn't the assumption that we check replica state which wouldn't be possible here?

This is a pre-flight check at the co-ordinator side, it's tracking large write operations, I added this because today we do have a pre-check on large read operations, and we take actions against those calls, should we have a similar implementation for write too?

Additionally, I have implemented replica side write threshold check too.

we do have a pre-check on large read operations

are those not different checks? There is read_size which is replica, and coordinator_read_size which is how much the coordinator used; this check has 2 very different definitions and won't be clear from client point of view what happens if these checks actually return a warn

Looks like the co-ordinator side checks can be guardrailed for read and write both. Removing these changes from this PR,a s it needs another one.

dcapwell · 2026-01-12T20:17:12Z

src/java/org/apache/cassandra/db/MutationVerbHandler.java

+        Message<NoPayload> reply = respondTo.emptyResponse();
+        reply = MessageParams.addToMessage(reply);
+        MessagingService.instance().send(reply, respondToAddress);


nit: you can make this a single line

MessagingService.instance().send(MessageParams.addToMessage(respondTo.emptyResponse()), respondToAddress);

dcapwell · 2026-01-12T20:18:20Z

src/java/org/apache/cassandra/db/WriteThresholds.java

+        DataStorageSpec.LongBytesBound sizeWarnThreshold = DatabaseDescriptor.getWriteSizeWarnThreshold();
+        int tombstoneWarnThreshold = DatabaseDescriptor.getWriteTombstoneWarnThreshold();
+
+        if (sizeWarnThreshold == null && tombstoneWarnThreshold == 0)


tombstoneWarnThreshold == -1

dcapwell · 2026-01-12T20:20:31Z

src/java/org/apache/cassandra/db/WriteThresholds.java

+            Long currentSize = org.apache.cassandra.db.MessageParams.get(ParamType.WRITE_SIZE_WARN);
+            if (currentSize == null || currentSize < estimatedSize)
+            {
+                org.apache.cassandra.db.MessageParams.add(ParamType.WRITE_SIZE_WARN, estimatedSize);


can we import this class to make this easier to read?

dcapwell · 2026-01-12T20:22:20Z

src/java/org/apache/cassandra/db/WriteThresholds.java

+        long estimatedSize = cfs.topPartitions.topSizes().getEstimate(key);
+        long estimatedTombstones = cfs.topPartitions.topTombstones().getEstimate(key);
+
+        if (sizeWarnBytes > 0 && estimatedSize > sizeWarnBytes)


Suggested change

if (sizeWarnBytes > 0 && estimatedSize > sizeWarnBytes)

if (sizeWarnBytes != -1 && estimatedSize > sizeWarnBytes)

Should be consistent. -1 is the "not enabled" case, so should check for that

dcapwell · 2026-01-12T20:24:17Z

src/java/org/apache/cassandra/db/WriteThresholds.java

+                org.apache.cassandra.db.MessageParams.add(ParamType.WRITE_SIZE_WARN, estimatedSize);
+
+                TableMetadata meta = update.metadata();
+                String pk = meta.partitionKeyType.getString(key.getKey());


think toCQLString is better than getString.

dcapwell · 2026-01-12T20:24:40Z

src/java/org/apache/cassandra/db/WriteThresholds.java

+
+                TableMetadata meta = update.metadata();
+                String pk = meta.partitionKeyType.getString(key.getKey());
+                logger.warn("Write to {}.{} partition {} triggered size warning; " +


this could spam, can we use a NoSpamLogger and only log once a minute?

Yes, thanks for highlighting this!

dcapwell · 2026-01-12T20:24:56Z

src/java/org/apache/cassandra/db/WriteThresholds.java

+
+                TableMetadata meta = update.metadata();
+                String pk = meta.partitionKeyType.getString(key.getKey());
+                logger.warn("Write to {}.{} partition {} triggered tombstone warning; " +


this could spam, can we use a NoSpamLogger and log once a minute?

dcapwell · 2026-01-12T20:25:33Z

src/java/org/apache/cassandra/db/WriteThresholds.java

+                org.apache.cassandra.db.MessageParams.add(ParamType.WRITE_TOMBSTONE_WARN, (int) estimatedTombstones);
+
+                TableMetadata meta = update.metadata();
+                String pk = meta.partitionKeyType.getString(key.getKey());


same as above, toCQLString is prob best here

dcapwell · 2026-01-12T20:26:41Z

src/java/org/apache/cassandra/db/WriteThresholds.java

+                String pk = meta.partitionKeyType.getString(key.getKey());
+                logger.warn("Write to {}.{} partition {} triggered size warning; " +
+                            "estimated size is {} bytes, threshold is {} bytes (see write_size_warn_threshold)",
+                            meta.keyspace, meta.name, pk, estimatedSize, sizeWarnBytes);


FYI the toString for TableMetadata is keyspace.name so you could simplify this and remove {}.{} and just do {} and pass in meta. It will also do the correct quoting if needed

dcapwell · 2026-01-12T20:26:47Z

src/java/org/apache/cassandra/db/WriteThresholds.java

+                String pk = meta.partitionKeyType.getString(key.getKey());
+                logger.warn("Write to {}.{} partition {} triggered tombstone warning; " +
+                            "estimated tombstone count is {}, threshold is {} (see write_tombstone_warn_threshold)",
+                            meta.keyspace, meta.name, pk, estimatedTombstones, tombstoneWarnThreshold);


FYI the toString for TableMetadata is keyspace.name so you could simplify this and remove {}.{} and just do {} and pass in meta. It will also do the correct quoting if needed

dcapwell · 2026-01-12T20:27:35Z

src/java/org/apache/cassandra/db/WriteThresholds.java

+        DataStorageSpec.LongBytesBound sizeWarnThreshold = DatabaseDescriptor.getWriteSizeWarnThreshold();
+        int tombstoneWarnThreshold = DatabaseDescriptor.getWriteTombstoneWarnThreshold();
+
+        if (sizeWarnThreshold == null && tombstoneWarnThreshold == 0)


Suggested change

if (sizeWarnThreshold == null && tombstoneWarnThreshold == 0)

if (sizeWarnThreshold == null && tombstoneWarnThreshold == -1)

dcapwell · 2026-01-12T20:46:40Z

src/java/org/apache/cassandra/metrics/TopPartitionTracker.java

            {
-                top.pollLast();
+                TopPartition p = top.pollLast();
+                lookup.remove(p.key);


you remove from the lookup but you don't add?

My bad, missed the lookup.put() call. Thanks!

dcapwell · 2026-01-12T21:07:51Z

src/java/org/apache/cassandra/service/paxos/PaxosCommit.java

            if (!Paxos.isInRangeAndShouldProcess(from, agreed.update.partitionKey(), agreed.update.metadata(), false))
                return null;

+            WriteThresholds.checkWriteThresholds(agreed.update, agreed.update.partitionKey());


im not sure if this would actually work in practice

if (executeOnSelf) { ExecutorPlus executor = PAXOS_COMMIT_REQ.stage.executor(); if (async) executor.execute(this::executeOnSelf); else executor.maybeExecuteImmediately(this::executeOnSelf); }

in some code paths we do a blocking execute, in others we do it async and ignore when it completes

As discussed, we hold onto making these changes in this patch.

dcapwell · 2026-01-12T21:12:38Z

src/java/org/apache/cassandra/service/writes/thresholds/CoordinatorWriteWarnings.java

+ * Accumulates warnings from multiple write operations in a single client request,
+ * then sends them to the client and updates metrics.
+ */
+public class CoordinatorWriteWarnings


there is a lot of overlap with CoordinatorWarnings, can we refactor so read/write can use 80% the same logic and only the read/write related changes need to be custom?

dcapwell · 2026-01-12T21:17:06Z

src/java/org/apache/cassandra/service/writes/thresholds/WarnCounter.java

+        maxWarningValue.accumulateAndGet(value, Math::max);
+        warnings.add(from);


this isn't thread safe... what is the concurrency expecations for this method?

It follows the same concurrency pattern as the read flow's present WarnAbortCounter.

Concurrency expectation:

Writers: Multiple n/w callback threads calling it concurrently.

Reader: Single coordinator thread calling concurrently with writers.

According to me the thread safety approach would be like update value first and add instance second.

This would basically ensure the ThresholdCounter handles the race condition correctly for read and write flows both, as the comment says:
call add last so concurrent reads see empty even if values > 0; if done in different order then size=1 could have values == 0

I can add this comment for Write flow to make this ordering explicit. Would that address your concern, or did you have a different concurrency scenario in mind ?

please add the following comment, which is from the read side

// call add last so concurrent reads see empty even if values > 0; if done in different order then // size=1 could have values == 0

When someone reads this code it can come off as a bug; without a comment explaining then future readers may break this without understanding it.

dcapwell · 2026-01-23T23:11:50Z

src/java/org/apache/cassandra/db/WriteThresholds.java

+/**
+ * Utility class for checking write threshold warnings on replicas.
+ */
+public class WriteThresholds


paxos / accord were left out of this patch intentionally. Can you leave a comment in this class to denote that?

something like

// CASSANDRA-17258: paxos and accord do complex thread hand off and custom write logic which makes this patch complex, so was deferred

dcapwell · 2026-01-23T23:18:00Z

src/java/org/apache/cassandra/service/AbstractWriteResponseHandler.java

+                    }
+                    else
+                    {
+                        logger.warn("Cannot propagate write warnings to coordinator because hintOnFailure is null. " +


if you use grep or intellij indexes to find where this comes from you will see BatchlogResponseHandler makes this null; which isn't a programming error. The field also annotates

private @Nullable final Supplier<Mutation> hintOnFailure;

denoted that this is in fact expected behavior

logger.warn can spam and flood operator logs, be very careful of this in the hot path

null looks like expected behavior for batch log, so not being able to update is fine

Sure, will keep an eye on it hence forth and update this as well.

dcapwell · 2026-01-23T23:20:14Z

src/java/org/apache/cassandra/db/WriteThresholds.java

+        long estimatedSize = cfs.topPartitions.topSizes().getEstimate(key);
+        long estimatedTombstones = cfs.topPartitions.topTombstones().getEstimate(key);
+        TableMetadata meta = update.metadata();
+        String pk = meta.partitionKeyType.toCQLString(key.getKey());


this is a costly operation and happens for every update, we should avoid creating this string unless we need to; which means at least one warning happened

a common pattern for this is to have String pk = null and w/e you enter the if block you check if null and compute it, that way you do it once and only do it if needed

Makes sense, let me change that!

dcapwell · 2026-01-23T23:22:22Z

Minor Issues & Nitpicks

🔵 Style: Missing newline at end of several files

Severity: Minor
Location: Multiple files including CoordinatorWriteWarnings.java, WarnCounter.java, WriteWarningContext.java, etc.

The diff shows \ No newline at end of file for many new files. This should be fixed for consistency with the rest of the codebase.

dcapwell · 2026-01-23T23:23:35Z

test/conf/cassandra.yaml

 local_read_size_fail_threshold: 8192KiB
 row_index_read_size_warn_threshold: 4096KiB
 row_index_read_size_fail_threshold: 8192KiB
+write_size_warn_threshold: 4096KiB


don't you also need write_thresholds_enabled: true?

dcapwell · 2026-01-23T23:28:17Z

src/java/org/apache/cassandra/service/writes/thresholds/CoordinatorWriteWarnings.java

+            Pair<TableId, DecoratedKey> key = Pair.create(update.metadata().id, update.partitionKey());
+            warnings.merge(key, snapshot);


Suggested change

Pair<TableId, DecoratedKey> key = Pair.create(update.metadata().id, update.partitionKey());

warnings.merge(key, snapshot);

warnings.merge(update.metadata().id, update.partitionKey(), snapshot);

code is easier to read if you push creating the Pair into the merge function. This is also a leaky detail as your API exposes how the internal data is structured.

dcapwell · 2026-01-23T23:38:34Z

src/java/org/apache/cassandra/service/writes/thresholds/CoordinatorWriteWarnings.java

+            }
+
+            TableMetadata metadata = cfs.metadata();
+            String partitionKey = metadata.partitionKeyType.getString(key.right.getKey());


Suggested change

String partitionKey = metadata.partitionKeyType.getString(key.right.getKey());

String partitionKey = metadata.partitionKeyType.toCQLString(key.right.getKey());

getString is almost never what you want sadly; toCQLString is.

Okay, will make sure I use CQLString hence forth!

dcapwell · 2026-01-23T23:40:30Z

src/java/org/apache/cassandra/config/Config.java


+    public volatile boolean write_thresholds_enabled = false;
+    public volatile DataStorageSpec.LongBytesBound write_size_warn_threshold = null;
+    public volatile int write_tombstone_warn_threshold = -1;


i don't see this validated in DatabaseDescriptor.apply can you make sure we validate? -1, 0+ seem to be the only valid values, but where is this enforced?

take a look at applyReadThresholdsValidations. Prob best to just rename that method and put the check there =)

For write_size_warn_threshold:
The validation is enforced by the DataStorageSpec type itself and the constructor validates that any non-null value must be >= 0. For this type, null represents "disabled" (equivalent to -1 for int fields). So technically the valid values are either null OR >= 0

I think it will be a redundant validation if I add soemthing similar to read thresholds, what do you think ?

dcapwell reviewed Jan 12, 2026

View reviewed changes

minal-kyada force-pushed the cassandra-17258 branch from 19d42eb to e145c0b Compare January 12, 2026 20:13

dcapwell reviewed Jan 12, 2026

View reviewed changes

minal-kyada force-pushed the cassandra-17258 branch 2 times, most recently from 803344e to d49ca1c Compare January 16, 2026 23:11

minal-kyada added 8 commits January 16, 2026 15:15

Config updates

bc5dc05

Network layer updates

1495b84

Write threshold check on replica

317174a

co-ordinator side aggregation

2b8b958

added metrics

0cc38ca

write threshold flag changes

865a849

co-ordinator side warn threshold pre-flight check

ed9cd52

Reformat DatabaseDescriptors file

7052758

Review comments addressed

c6951fb

minal-kyada force-pushed the cassandra-17258 branch from d49ca1c to c6951fb Compare January 16, 2026 23:19

Unite tests added

66c4f9c

minal-kyada force-pushed the cassandra-17258 branch from d7d15d4 to 331fbc7 Compare January 23, 2026 20:43

dcapwell reviewed Jan 23, 2026

View reviewed changes

Added Distributed tests

0763316

minal-kyada force-pushed the cassandra-17258 branch from 331fbc7 to 0763316 Compare January 24, 2026 02:04

		maybeWarnWriteSize(mutations, options);
		maybeWarnWriteTombstones(mutations, options);

	if (sizeWarnBytes > 0 && estimatedSize > sizeWarnBytes)
	if (sizeWarnBytes != -1 && estimatedSize > sizeWarnBytes)

	if (sizeWarnThreshold == null && tombstoneWarnThreshold == 0)
	if (sizeWarnThreshold == null && tombstoneWarnThreshold == -1)

		maxWarningValue.accumulateAndGet(value, Math::max);
		warnings.add(from);

		Pair<TableId, DecoratedKey> key = Pair.create(update.metadata().id, update.partitionKey());
		warnings.merge(key, snapshot);

	Pair<TableId, DecoratedKey> key = Pair.create(update.metadata().id, update.partitionKey());
	warnings.merge(key, snapshot);
	warnings.merge(update.metadata().id, update.partitionKey(), snapshot);

	String partitionKey = metadata.partitionKeyType.getString(key.right.getKey());
	String partitionKey = metadata.partitionKeyType.toCQLString(key.right.getKey());

CASSANDRA-17258: Add write threshold warnings for large partitions and tombstones #4556

Are you sure you want to change the base?

CASSANDRA-17258: Add write threshold warnings for large partitions and tombstones #4556

Conversation

minal-kyada commented Jan 12, 2026

Summary

Related Work

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

minal-kyada Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dcapwell commented Jan 23, 2026

Minor Issues & Nitpicks

🔵 Style: Missing newline at end of several files

Uh oh!

minal-kyada Jan 16, 2026 •

edited

Loading