2

Is it possible to have several brokers behind a single TCP/IP port?

I.e., offer a single endpoint for Kafka clients, and having a scalable amount of brokers behind that?

In my company, some people are configuring a (scaling) Kafka server for us, and they insist that this is not possible, and that we need to configure every single broker as a separate endpoint...

This is in a OpenShift context; it would be nice to just be able to scale the Kafka pods up and down as needed, while using the usual loadbalancing to offer a single port to the outside world.

EDIT: I am familiar with HAProxy; the question is about Kafka, not OpenShift/ HAProxy. Is there anything special to be configured (in the Kafka broker image)

  • Are they all synchronized amongst each other?
  • Are there examples for that somewhere?

Googling for "OpenShift Apache Kafka" etc. brings up plenty of examples, but all that I found seem to have a different service for each Apache Kafka broker (i.e., explicitly running 3 separate Kafka deployments inside OpenShift, with separate OpenShift services, one each). This is what I wish to avoid. I want to horizontally scale the number of Kafka processes behind HAProxy, and have one single external IP.

Tensibai
  • 11,416
  • 2
  • 37
  • 63
AnoE
  • 4,936
  • 14
  • 26

1 Answers1

1

Yes, you can. HAProxy is purpose built to balance any TCP or HTTP connection. My old company used to run every component of Kafka with HAproxy.

Here is a whole project dedicated to running auto-scaling Kafka in Docker using HAproxy: https://github.com/pozgo/docker-kafka

You can even see their HAproxy config here where they specify the broker check (port 9092): https://github.com/pozgo/docker-kafka/blob/master/haproxy.cfg

BoomShadow
  • 1,472
  • 1
  • 15
  • 11