Multiple kafka brokers

[coolwednesday]

Pull Request Template

Description:

• Closes How to configure Gofr for Kafka with multi brokers cluster #1550
• This PR handles connection to multiple kafka brokers provided as comma seperated values in .env file.
• Please note we do not actually handle creation of topic in a specific broker. Depending on the availibility, Kafka does that itself.
• HealthCheck now returns information about all the open broker connections.
• To test, we can set up three kafka brokers using the following docker-compose.yml :

version: '3'
services:
 zookeeper:
   image: confluentinc/cp-zookeeper:7.2.1
   container_name: zookeeper
   environment:
     ZOOKEEPER_CLIENT_PORT: 2181
 kafka1:
   image: confluentinc/cp-kafka:7.2.1
   container_name: kafka1
   ports:
     - "8097:8097"
   depends_on:
     - zookeeper
   environment:
     KAFKA_BROKER_ID: 1
     KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
     KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: EXTERNAL:PLAINTEXT,INTERNAL:PLAINTEXT
     KAFKA_ADVERTISED_LISTENERS: EXTERNAL://localhost:8097,INTERNAL://kafka1:9092
     KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
 kafka2:
   image: confluentinc/cp-kafka:7.2.1
   container_name: kafka2
   ports:
     - "8098:8098"
   depends_on:
     - zookeeper
   environment:
     KAFKA_BROKER_ID: 2
     KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
     KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: EXTERNAL:PLAINTEXT,INTERNAL:PLAINTEXT
     KAFKA_ADVERTISED_LISTENERS: EXTERNAL://localhost:8098,INTERNAL://kafka2:9092
     KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
 kafka3:
   image: confluentinc/cp-kafka:7.2.1
   container_name: kafka3
   ports:
     - "8099:8099"
   depends_on:
     - zookeeper
   environment:
     KAFKA_BROKER_ID: 3
     KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
     KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: EXTERNAL:PLAINTEXT,INTERNAL:PLAINTEXT
     KAFKA_ADVERTISED_LISTENERS: EXTERNAL://localhost:8099,INTERNAL://kafka3:9092
     KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL

• Then specify the host address of the brokers in pubsub examples present in the examples folder.

Checklist:

• I have formatted my code using goimport and golangci-lint.
• All new code is covered by unit tests.
• This PR does not decrease the overall code coverage.
• I have reviewed the code comments and documentation for clarity.

Thank you for your contribution!

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[ccoVeille]

Except the constant that could be fixed now.

My refactoring suggestion could be addressed in another PR.

The code is too complex and could be simplified by using kafkaClient method and not standalone functions