Server Fault
Server Fault

Questions tagged [kafka]

Kafka is a distributed messaging queue application written in java.

Kafka website: http://kafka.apache.org/

When asking a question about provides relevant information like the kafka version and the version. The design of your cluster should be mentioned with the number of brokers, the topic configuration.

When you have a bug on a particular topic, perform the same test on another topic with different parameters and observe if you can reproduce.

109 questions
17
votes
9 answers

Unable to start kafka with zookeeper (kafka.common.InconsistentClusterIdException)

Below the steps I did to get this issue : Launch ZooKeeper Launch Kafka : .\bin\windows\kafka-server-start.bat .\config\server.properties Also note that my OS is Windows And at the second step the error happens : ERROR Fatal error during…
TourEiffel
  • 271
  • 1
  • 2
  • 5
8
votes
3 answers

Kafka logfile location

Just to make some things clear: I'm talking about the process logfile that contains the stdout and stderr messages. This is my systemd unit file: [Unit] Description=Apache Kafka server Documentation=http://kafka.apache.org Requires=network.target…
Jeroen Jacobs
  • 1,436
4
votes
1 answer

kafka failed authentication due to: SSL handshake failed

I have to add encryption and authentication with SSL in kafka. This is what I have done: - 1) Generate certificate for each broker kafka: COMANDO: keytool -keystore server.keystore.jks -alias localhost -validity 365 -genkey - 2) Create CA. The…
xRobot
  • 141
4
votes
1 answer

Kafka compatible with Zookeeper 3.5 feature 'Rebalancing Client Connections'

In this document https://zookeeper.apache.org/doc/trunk/zookeeperReconfig.html dynamic configuration functionality is described for Zookeeper 3.5. There are 2 important points in this document: When changing the dynamic config of one Zookeeper…
hendrikswan
  • 141
  • 5
4
votes
1 answer

Choose number of partitions in Kafka

I read the documentation of apache kafka but I couldn't find an example about how many partitions should I use in any scenario. For example lets say that I have 5000 msgs/entries per minute, for this situation how many partitions should I have (or…
Diego Velez
  • 835
4
votes
0 answers

Kafka using a distant Zookeeper instance?

we have an existing Zookeeper instance, and want to build a Kafka cluster in a remote location (not on the same side of the Atlantic ocean). Can we use that existing Zookeeper for our new Kafka usage ? Would latency be an issue ? Kafka is using…
Pixou
  • 141
4
votes
1 answer

Kafka stores log4j logs in directory literally called ${kafka.logs.dir}

I installed a preview version of Kafka 3.0 with log4j2 support (http://home.apache.org/~dongjin/post/apache-kafka-log4j2-support/) on a RHEL 8 server. Kafka and Zookeeper are running successfully as systemd user services. I set the environment…
Cabbage Parachute
  • 43
  • 1
  • 6
3
votes
2 answers

Enable logical replication on Google Cloud Postgres

Is there anyway to enable logical replication wal_level = logical on Google Cloud Postgres instance? I want to do CDC (change data capture) to stream WAL Record to Apache Kafka. Thanks for your help!
Kevin
  • 133
3
votes
1 answer

Does Kafka replicate the same data across all brokers?

I am going to implement Kafka in the infrastructure. I have a use case where we have below components in the Kafka architecture. Number of brokers: 5 Replication Factors: 3 Number of topics: 3 Number of partition: 3 I have to implement scaling…
mohit
  • 221
3
votes
2 answers

Apache Kafka configuration behind NAT/LB

In order to make our Kafka cluster available both from Internet and from our private network, we thought fine to configure Kafka this way : Private VIP:9000 => All brokers:9092 (topology query only) Private VIP:9001 => Broker #1:9092 Private…
mveroone
  • 484
3
votes
2 answers

What is ideal directory for kafka messages

The default value of logs.dir=/tmp/kafka-logsin server.properties. Usually /tmp is avoided from keeping any important files and we are storing messages and offsets! Any particular reason why one may not choose /var/log/kafka-logs or…
Divs
  • 131
3
votes
1 answer

Running Kafka in Kubernetes on AWS

I've the following scenario: 3 nodes instances spread across 3 Availability Zones 6 Brokers running in StatefulSets with AWS external volume for Kafka data. Instance size: m4.2xlarge EBS Volume: st1 - 500 GiB No limit requests and memory set at…
Mazzy
  • 257
3
votes
1 answer

How to read back collectd and write_kafka plugin

I want to use kafka as a transport layer for collectd. I found that there is a write_kafka plugin for collectd, which sends all the gathered metrics to a kafka topic. My intention was to have a few hosts as collectors (working as a kafka consumer…
Jeroen Jacobs
  • 1,436
3
votes
0 answers

why is my kafka cluster under such a high load for only 1.2K clients, 3K message/s, max bandwidth 25Mb/s

I have a single cloud server, the spec: Intel Xeon 4 core 3GHz Cpu, 16G memory I've run a kafka cluster(3 kafka instances + zookeeper) via docker compose on it(docker images were pulled at early 2024) with all latest tag for docker images, one of…
Shawn
  • 141
3
votes
1 answer

Page cache in Kafka

We are receiving alerts for kafka memory utilization in our kubernetes cluster and analysing it seems that most of the memory is page cache. I am aware that kafka uses loads of page cache for IO performance and reads/writes are cached before…
Putharekulu
  • 41
1
2 3 4 5 6 7 8