[Bug]: Transactional handling of resolution of multiple resources managing the same topic

[rolandjohann]

Bug Description

When two resources manage the same topic, the second resource will be handled as conflict by the topic operator. Deleting the first resource leads to deletion of the topic and at the next reconciliation run the second resource seems to be evaluated again, the conflict is not present anymore and the topic will be recreated.

Question is if this is desired behavior or if we can improve it to be transactional by checking if conflicting resources are present on first resource delete and not perform the actual topic delete at kafka then.

Cause of this use case is that we hand over the ownership of the topic resource to another team, also at kubernetes level, which resulted in topic deletion and recreation.

Steps to reproduce

1. watch kafka topics $ watch './bin/kafka-topics.sh [...] --list | grep poc-test-topic'
2. apply kafka topic resource and wait for creation

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaTopic
   metadata:
     labels:
       strimzi.io/cluster: poc-kafka
     name: poc-test-topic
   spec:
     topicName: poc-test-topic
     partitions: 1
     replicas: 3

3. apply second kafka topic

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaTopic
   metadata:
     labels:
       strimzi.io/cluster: poc-kafka
     name: poc-test-topic-different-but-same
   spec:
     topicName: poc-test-topic
     partitions: 1
     replicas: 3

4. Topic operator logs warning

   KafkaTopic([...]/poc-test-topic-different-but-same): Reconciliation failed: Managed by Ref{namespace='[...]', name='poc-test-topic'}

5. topic poc-test-topic is present
6. delete kafka topic resource poc-test-topic
7. output of $ watch ... shows that kafka topic has been deleted at kafka
8. kafka topic resource poc-test-topic-different-but-same shows state is not ready
9. reconciliation of topic operator kicks in
10. topic is created at kafka and visible at $watch ... again

Expected behavior

1. watch kafka topics $ watch './bin/kafka-topics.sh [...] --list | grep poc-test-topic'
2. apply kafka topic resource and wait for creation (first resource, see above)
3. apply second kafka topic (second resource, see above)
4. Topic operator logs warning

   KafkaTopic([...]/poc-test-topic-different-but-same): Reconciliation failed: Managed by Ref{namespace='[...]', name='poc-test-topic'}

5. topic poc-test-topic is present
6. delete kafka topic resource poc-test-topic
7. output of $ watch ... still shows the topic since it is not deleted
8. kafka topic resource poc-test-topic-different-but-same shows state to be ready
9. reconciliation of topic operator kicks in, nothing to reconcile
10. topic is not deleted and recreated

Strimzi version

0.48.0

Kubernetes version

v1.32.5

Installation method

Helm chart

Infrastructure

Azure AKS

Configuration files and logs

No response

Additional context

No response

[fvaleri]

Hello @rolandjohann. You can use the unmanage annotation to delete a KafkaTopic custom resource without deleting the actual topic in Kafka. After that, the other KafkaTopic will be reconciled without topic recreation.

https://strimzi.io/docs/operators/latest/deploying#proc-converting-managed-topics-str

[rolandjohann]

Hi @fvaleri thanks for the quick reply. Sure, there are manual ways to control orchestration/automation behavior of the different operators, but that was not the question.

In question is the desired behavior of the topic operator in the case described originally. It boils down to the assumption that when there are kubernetes resources present defining a topic, there is a certain edge case where the topic will be deleted and recreated after some time.

[fvaleri]

I would say this is the expected behavior, and the edge case is caused by a misconfiguration. Having multiple KafkaTopic resources managing the same Kafka topic violates the principle of single ownership. Transactional checks across multiple resources would add significant complexity for very little benefit IMO.

[scholzj]

I would agree with Federico. I would not go as far as say it is desired - the whole situation here is undesired. But from all the options, this seems to me like the most desired one. Although I'm sure different people might have different views on this and it is not completely black-and-white.

[rolandjohann]

@scholzj Completely agree - we treated it as undefined behavior until we tested it. I also understand that you don't want to touch that topic at the implementation.

Shall we add a hint to the docs that those conflicts won't resolve cleanly?

[scholzj]

Sure, why not - we can definitely try to improve the docs about it. Feel free to open a PR or an issue for it. Thanks.