Software Engineering Stack Exchange
Software Engineering Stack Exchange

Questions tagged [apache-kafka]

Apache Kafka is publish-subscribe messaging rethought as a distributed commit log.

Apache Kafka is publish-subscribe messaging rethought as a distributed commit log.

98 questions
17
votes
2 answers

Traditional Message Brokers and Streaming Data

According to the Kafka site: "Kakfa is used for building real-time data pipelines and streaming apps." Searching the internet far and wide, I've found the following generally-accepted definition of what "stream data" is: Stream data is data that…
smeeb
  • 4,950
  • 10
  • 33
  • 52
11
votes
2 answers

Event sourcing, replaying and versioning

I am designing a system that uses Event Sourcing, CQRS and microservices. I am lead to understand this isn't an uncommon pattern. A key feature of the service needs to be the ability to rehydrate/restore from a system of record. Microservices will…
Antony Woods
  • 353
  • 1
  • 3
  • 11
10
votes
1 answer

Data durability guarantees in Kafka

Is it wise to use kafka as the 'source of truth' for mission critical data? The setup is: kafka is the underlying source-of-truth for the data. -querying is done on caches (I.e. Redis, ktables) hydrated from kafka Kafka configured for durability…
major_clanger
  • 111
8
votes
2 answers

Is Kafka needed in a realtime chat application?

I'm developing a realtime chat application with an Angular frontend and Java backend. I've found a couple of examples that resemble what I am trying to achieve, such…
Babyburger
  • 221
8
votes
1 answer

Is it appropriate to say that RabbitMQ and Apache Kafka solve similar problems?

I have used RabbitMQ but I haven't used Apache Kafka. Is it a similar problem that these products solve, or is there no connection?
Niklas Rosencrantz
  • 8,178
8
votes
3 answers

Decoupling microservices with gRPC

I'm setting up a microservices architecture, and am confused about how gRPC can loosely-couple services (compared to a pub-sub message service like Kafka). Doesn't the request go directly to the server, and not through a pub/sub system? While gRPC…
skunkwerk
  • 189
6
votes
1 answer

Can Event Sourcing and an API Management platform leverage each other?

I'm evaluating the use of Kafka to implement Event Sourcing in a microservices environment that already makes use of an API Management platform. I was wondering if it is common and/or considered a good practice to use the API Management as a gateway…
ivarec
  • 315
  • 1
  • 6
5
votes
2 answers

kafka consumer properties session.timeout.ms vs heartbeat.interval.ms

I am reading Kafka, the definitive guide and I came across the below point for consumers. heartbeat.interval.ms must be lower than session.timeout.ms, and is usually set to one-third of the timeout value. May I know why is it, I thought both should…
Ramya B
  • 59
4
votes
3 answers

Alternating between Java streams and parallel streams at runtime

This is a question I constantly ask myself when designing a data intensive application: When is it appropriate to use stream() over parallelStream()? Would it make sense to use both? How do I quantify the metrics and conditions to intelligently…
user0000001
  • 263
4
votes
2 answers

Data processing pipeline design for processing data

I have a use case for which I need to build a data processing pipeline Customer contact leads data coming from different data sources like csv, data base, api has to be first mapped to a universal schema fields. There could be ~100k rows coming…
Sam Rohn
  • 149
4
votes
2 answers

Event Driven Microservice Race Condition

Say we have 4 services, A through D, which communicate (for the most part) through some sort of asynchronous event-driven system. When a new entity is created in A, B & C receive that event. B creates an entity of its own based on that event, and C…
Asmodean
  • 149
4
votes
2 answers

Two processes in a single docker container or two services connecting to the same db?

I recently started moving a monolithic application to microservices architecture using docker containers. The general idea of the app is: scraping data -> format the data -> save the data to MySQL -> serve data via REST API. I want to split each…
rogamba
  • 151
3
votes
1 answer

Deciding on granularity of SOLID's SRP

I am currently working in a project where we are utilizing kafka as a Message Queue. We have two use cases here, one is to consume the messages in parallel, no ordering of any kind required. And one is to have ordering based on partitioning key…
Akshansh Jain
  • 139
3
votes
2 answers

In Hexagonal Architecture how do I implement a Dead-Letter Queue?

I have an application using Hexagonal Architecture. I want to implement a DLQ in 2 scenarios: When the event streaming adapter's incoming message is unreadable (invalid payload, etc) When the domain service identifies the…
Ryan
  • 133
3
votes
2 answers

Microservice architecture pattern for Batch based system

I have been exploring the microservice architecture for the batch-based system. Here is our current setup: Code: We have 5 systems that are internally connected and they pass data from one system to another. Currently entire logic is sitting in…
SMaZ
  • 89
1
2 3 4 5 6 7