Add support for modifying topic replication count

[modax]

Expected Behavior

Topic replica count to be changed if modification is allowed for the topic as per Configuring Topics.

I'm aware that the task is not trivial or straightforward. However, you can get quite some inspiration from the corresponding kafkactl functionality.

Current Behavior

As far as I can tell, org.springframework.kafka.core.KafkaAdmin#addOrModifyTopicsIfNeeded() does not support changing topic replication count. KafkaAdminClient API alterPartitionReassignments() needs to be invoked for this to happen.

Context

I expected this to be supported if spring-kafka is to be used for topic management.

[sobychacko]

@modax Since the underlying KafkaAdmin operation does not support that feature fundamentally, I think any solution that comes up with in the framework will be somewhat custom. However, we can take a look at this and see if there is a way to support this. But, I am not sure when we can get to it, if you have ideas, feel free to work on a POC.

[chickenchickenlove]

Hi. @modax , @sobychacko

Would it be okay for me to take a crack?

[sobychacko]

Sure, please go ahead.

[modax]

@chickenchickenlove feel free!

[chickenchickenlove]

@modax @sobychacko

I have no idea about the scenario that spring-kafka modifies replication count of existing topic.
kafkactl can do it, because it is ctl. so users will use it when they need to do it.

However, spring-kafka is framework, not a ctl.
IMHO, If spring-kafka users want to modify replicas of existing topic, it would be better for users to use kafkactl or kafka-reassign-partitions.sh for modifying replicas instead of spring-kafka considering that spring-kafka is framework.
As you know, re-built replicas in other broker will take long time even if there are more topic partitions, replicas.

I have no idea about the scenario that spring-kafka modifies replication count of existing topic.
Could you let me know the scenario you imagined before?
Or, Could you please suggest use-case about this.

If so, I'd be happy to consider implementing something tailored to your use case! 🙇‍♂

[modax]

Well, I have encountered this when researching spring-kafka as a way to manage topics. At first I thought it was only able to create new topics. Then I saw it was able to modify topics as well (for example increase partition count). And then saw that this aspect is missing (I mean replicas(X) of NewTopic might get out of date). To be honest, I don't see much difference between kafkactl and spring-kafka with regards to topic management since both already have (some of) it. Just that in case of spring-kafka, it is "closer to the application code".

Now to be honest I don't know the specifics of what it takes for Kafka broker to reassign partitions. In particular, if this can be done a topic being used at the time, how long it takes for replica count to be increased (make sense that synchronization might take some time) or decreased (should be quick, shouldn't it?). As far as I can tell, kafkactl algorithm is pretty smart. It does not reassign existing partitions which are going to be needed anway, but only adds or removes them to match replica count. It also avoids adding multiple partition replicas to the same broker (as far as I can tell anyway).