0

I want to create Kafka cluster in KRaft mode with two controllers and two brokers. I wrote configuration:

services:

kafka-controller-1:
    image: 'bitnami/kafka:latest'
    environment:
      - KAFKA_CFG_NODE_ID=0
      - KAFKA_KRAFT_CLUSTER_ID='4ea7fcb4a2fb4dc8a402d4103ee414eb'
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@kafka-controller-1:9092,1@kafka-controller-2:9092
      - KAFKA_CFG_PROCESS_ROLES=controller


  - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
  - KAFKA_CFG_LISTENERS=CONTROLLER://kafka-controller-1:9092
  - KAFKA_CFG_ADVERTISED_LISTENERS=CONTROLLER://kafka-controller-1:9092
  - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT




kafka-controller-2:
    image: 'bitnami/kafka:latest'
    environment:
      - KAFKA_CFG_NODE_ID=1
      - KAFKA_KRAFT_CLUSTER_ID='4ea7fcb4a2fb4dc8a402d4103ee414eb'
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@kafka-controller-1:9092,1@kafka-controller-2:9092
      - KAFKA_CFG_PROCESS_ROLES=controller


  - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
  - KAFKA_CFG_LISTENERS=CONTROLLER://kafka-controller-2:9092
  - KAFKA_CFG_ADVERTISED_LISTENERS=CONTROLLER://kafka-controller-2:9092
  - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT




kafka-broker-1:
    image: 'bitnami/kafka:latest'
    environment:
      - KAFKA_CFG_NODE_ID=2
      - KAFKA_KRAFT_CLUSTER_ID='4ea7fcb4a2fb4dc8a402d4103ee414eb'
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@kafka-controller-1:9092,1@kafka-controller-2:9092
      - KAFKA_CFG_PROCESS_ROLES=broker


  - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
  - KAFKA_CFG_LISTENERS=PLAINTEXT://kafka-broker-1:9092
  - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT




kafka-broker-2:
    image: 'bitnami/kafka:latest'
    environment:
      - KAFKA_CFG_NODE_ID=3
      - KAFKA_KRAFT_CLUSTER_ID='4ea7fcb4a2fb4dc8a402d4103ee414eb'
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@kafka-controller-1:9092,1@kafka-controller-2:9092
      - KAFKA_CFG_PROCESS_ROLES=broker


  - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
  - KAFKA_CFG_LISTENERS=PLAINTEXT://kafka-broker-2:9092
  - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT




kafka-ui:
    container_name: kafka-ui
    image: 'provectuslabs/kafka-ui:latest'
    environment:
      - KAFKA_CLUSTERS_0_NAME=test-local
      - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka-broker-1:9092
    depends_on:
      - kafka-controller-1
      - kafka-controller-2
      - kafka-broker-1
      - kafka-broker-2
    ports:


    

  • '1354:8080'
    •

It works, UI shows me two brokers and one of them a controller. (I can't post image, need 10 reputation)

My questions is:

  1. Why I see only two brokers and no one of my controllers?
  2. Why one of brokers is controller if i configure other two controllers?

Additional:

  1. - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER configured for brokers because without this I get error:

    Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: controller.listener.names must contain at least one value when running KRaft with just the broker role

  2. If I connect UI to controller - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka-controller-1:9092 that UI can't request METADATA:

    2025-02-08 00:18:21 2025-02-07 21:18:21,979 INFO [kafka-admin-client-thread | kafka-ui-admin-1738963090-1] o.a.k.c.a.i.AdminMetadataManager: [AdminClient clientId=kafka-ui-admin-1738963090-1] Metadata update failed 2025-02-08 00:18:21 org.apache.kafka.common.errors.UnsupportedVersionException: The broker does not support METADATA

Pedro
  • 1

0 Answers0