Why Kafka?-I
Multiple Producers
Kafka is able to seamlessly handle multiple producers, whether those clients are using
many topics or the same topic. This makes the system ideal for aggregating data from
many frontend systems and making it consistent. For example, a site that serves content
to users via a number of microservices can have a single topic for page views that
all services can write to using a common format. Consumer applications can then
receive a single stream of page views for all applications on the site without having to
coordinate consuming from multiple topics, one for each application.
Multiple Consumers
In addition to multiple producers, Kafka is designed for multiple consumers to read
any single stream of messages without interfering with each other. This is in contrast
to many queuing systems where once a message is consumed by one client, it is not
available to any other. Multiple Kafka consumers can choose to operate as part of a
group and share a stream, assuring that the entire group processes a given message
only once.
Disk-Based Retention
Not only can Kafka handle multiple consumers, but durable message retention means
that consumers do not always need to work in real time. Messages are committed to
disk, and will be stored with configurable retention rules. These options can be
selected on a per-topic basis, allowing for different streams of messages to have different
amounts of retention depending on the consumer needs. Durable retention
means that if a consumer falls behind, either due to slow processing or a burst in traffic,
there is no danger of losing data. It also means that maintenance can be performed
on consumers, taking applications offline for a short period of time, with no
concern about messages backing up on the producer or getting lost. Consumers can
be stopped, and the messages will be retained in Kafka. This allows them to restart
and pick up processing messages where they left off with no data loss.