Why is zookeeper.connect a forbidden config?

[jeroenmoons-klarrio]

I am looking into making the migration from a koperator-managed Kafka cluster to Strimzi.

Without going into too much detail, I would really like the Strimzi Kafka to work with our existing ZooKeeper, rather than Strimzi rolling its own and me having to migrate the ZK data from the old to the new setup. The old setup also has all metadata under a 'subdir' '/kafka/[CLUSTER_NAME], making it harder to pick that data up under Strimzi since the ZK chroot can only be set through that same zookeeper.connect` parameter.

It seems like the zookeeper.connect setting is protected only by its presence in FORBIDDEN_PREFIXES. So it would seem quite straightforward to leave zookeeper.connect (and zookeeper.ssl) out of that list and configure Strimzi to talk to my existing ZK instead. That would save me a lot of headache.

My question: what is the reason Strimzi protects this configuration?

The scenario I am considering is:

• Fork Strimzi to make zookeeper.connect an allowed setting
• Do my migration plan to bring the brokers under Strimzi-management while keeping my current ZK
• Then leverage the Strimzi ZK to KRaft migration feature
• Once done, ZK is out of the picture and I can switch back to the official Strimzi release

Would it be as simple as allowing that configuration, or would that break any assumptions Strimzi makes and cause trouble somehow? If during this migration, Strimzi brings up a separate ZooKeeper because it thinks it has to, that does not matter in the slightest as long as the brokers talk to mine. I just need to do what needs to be done to get to a Strimzi + KRaft situation and if the journey itself is ugly that does not matter much.

The way I see it, a Kafka broker just needs to know where ZooKeeper is and how to talk to it. If I force Strimzi to allow me setting the ZooKeeper connection configuration (should also include SSL I realize now) that should be enough to have a Strimzi-Kafka with a BYO ZooKeeper.

Any insights here would be greatly appreciated.

There is precious little information to find on koperator to Strimzi migrations :-) I am open to suggestions there as well. MY requirements state I should swap koperator for Strimzi without consumers noticing any change: that includes keeping consumer offsets identical (ruling out MirrorMaker).

[scholzj]

To be honest, I do not know if anyone can give you a simple answer. And I'm not aware of anyone ever trying this.

Strimzi is designed to use its own ZooKeeper cluster. It is part of Strimzi from the beginning. You cannot connect it to any other ZooKeeper. Even if you unblock these options, Strimzi would still deploy its own ZooKeeper cluster. And I'm afraid I'm not sure whether it would or would not be looking for some Kafka-related information there, so even if we ignore the running unused ZooKeeper cluster, I'm not sure it would work.

I think it is similar with the migration. It might work, but there migth be also situations when the operator needs to touch the rigth ZooKeeper and will be missing information. I think there are at least some ZooKeeper path deletions during the migration rollback etc. @ppatierno might know how much the migration really touches the operator.

So I think you would need to try these things your self and see how well it works when you just modify the forbidden options and what other things would need to be modified.

But I'm not entirely sure how much it really helps you to migrate your cluster without the consumers noticing. Will you shut down the old Kafka brokers and copy the data manually to Strimzi brokers? I also guess there will be some address changes and similar things?

[ppatierno]

Regarding the KRaft migration, being stuck on Kafka 3.7.x could be an issue. Several migration bugs were fixed along the way, so the best start for migrating from ZK to KRaft is having Kafka 3.9.x running (which is supported by our Strimzi 0.45.1, the latest version with ZK and KRaft migration implementation). So it seems to me, unless I am missing something, that after having Strimzi using your ZK and before the KRaft migration, you should go through a Kafka upgrade to 3.9.x.

[scholzj]

I'm not sure there is a reason to be stuck with Kafka 3.7 in that case -> if the brokers are moved by re-using the disks, they can be likely reused with Kafka 3.9 brokers.

[jeroenmoons-klarrio]

Great further input, thanks a lot! It's going to be challenging that's for sure.

[jeroenmoons-klarrio]

A concern that just surfaced: we are using jbod storage with multiple dirs. Are we going to be running into trouble with that if we want to migrate to KRaft under Strimzi 0.45.1?

We found a few resources that talk about supporting this but status is unclear to us.

We found this proposal, for instance, but are not sure what is and is not supported at this point: https://github.com/strimzi/proposals/blob/main/067-kraft-jbod-support.md

[ppatierno]

Strimzi support JBOD storage for multiple disks since long time now https://strimzi.io/docs/operators/latest/deploying#assembly-storage-str