Every Kafka system has a producer that writes events and a consumer that reads them. They are just apps you already know.
Any app that has events to share: web servers (signups, clicks), mobile apps, backend services, IoT devices, microservices.
Any app that needs those events: analytics dashboards, notification services, ML pipelines, audit logs, search indexers.
Producers and consumers don't talk to each other directly. There's a middleman in between. That's what the next slide introduces.
...but where do messages go in between?
A broker is a server that stores messages between the producer and the consumer. The producer writes to it; the consumer reads from it.
A Kafka broker is a long-running server process. You install Kafka, start a broker, and now producers can write to it and consumers can read from it.
Files on disk. Messages land in append-only log files. They sit there until the retention period expires (default: 7 days).
A single broker is fine for dev. Production runs many brokers together — that's a cluster. More on that after we look at topics.
producer → broker → consumer — the producer never sees the consumer
The broker organizes messages into topics. A producer writes to a topic; a consumer reads from one. Think: a topic is a category of events.
Related events. Examples: user-signups, order-events, page-views, audit-log. Pick a name that describes the event type.
The producer picks the topic name when sending. The consumer subscribes to whichever topics it cares about. Both must agree on the name.
The same topic can be read by many consumers independently. Analytics, notifications, and ML can all read from order-events at their own pace.
A topic is split into partitions — ordered, append-only log files. Each partition lives on one broker, but a topic's partitions can be spread across the cluster.
More partitions = more parallel writers = higher throughput. One big file is a bottleneck; many smaller files write in parallel.
A topic is a name. The partitions under that name are files scattered across your cluster.
Plan partition count from your peak load, not your current load. You can add partitions, but you can never reduce them.
Click Add Record a few times. Records round-robin across the three partitions. Within one partition, offsets are strict.
A consumer group is a team sharing the work. Different groups read the same topic independently at their own pace.
Add consumers to a group → partitions rebalance → more parallel work. But: more active consumers than partitions = idle ones.
Group A (analytics) and Group B (notifications) read the same topic at their own pace. One crashing doesn't affect the other.
Add a 4th consumer to either group. It goes idle — you can never have more active consumers than partitions.
With a key, all events for one entity always land in the same partition — guaranteeing order for that entity.
Producer round-robins across partitions. User 1's events can land in P0, P2, P1 — order is lost for that user.
Producer hashes the key. All events with user_id=42 always go to the same partition — strictly ordered for that user.
Send User 1 three times in No Key mode — events scatter. Switch to User ID Key — all three cluster in one partition, in order.
Pick the key by what needs ordering — e.g. user_id, account_id, order_id. One key → one partition → strict order.
The log keeps appending forever — your disk doesn't. Two cleanup strategies, both running in the background: time retention deletes by age, log compaction keeps only the latest value per key.
A topic is an append-only log. Every record is appended, never updated. Without cleanup, the log grows forever until the disk fills up. You pick a cleanup policy per topic.
Default. Kafka deletes records older than N days (e.g. 7) or beyond a size limit. For long-term storage, sink to a database or data lake.
For "current state" topics (user profiles, config). Kafka keeps only the latest record per key. New consumers get the current state instantly — no replay needed.
Anything older than 5 days is deleted
The log now holds the current state: Alice = click, Bob = login.
When does cleanup happen? Continuously, in the background. As soon as a log segment is closed (default 1 GB or 7 days), the broker checks the topic's cleanup policy and removes eligible records. Producers and consumers never block on it.
Kafka remembers the recent past. For long-term history, sink to a database. For "current state only" topics, use log compaction.
RabbitMQ is a task board — take a sticky note, do the task, throw it away. Kafka is an archive — everyone can read the same record, and it stays put.
RabbitMQ = queue, delete on consume. Kafka = log, retain on consume. They are not the same category of tool.
Replay (rewind any consumer). Multi-consumer (groups read independently). Order at scale (per partition). Event sourcing. Throughput (millions/sec).
Simple background jobs (PDF, email, thumbnails). Rich routing. Low operational complexity. Workloads in the thousands, not millions.
| Dimension | RabbitMQ | Kafka |
|---|---|---|
| Model | Queue (delete on read) | Log (retain on read) |
| Throughput / node | ~10k–50k msg/s | ~100k–1M msg/s |
| Replay | No | Yes |
| Multi-consumer reading | Fanout exchange (extra setup) | Native — every group is independent |
| Routing | Rich (topic, headers, fanout) | Simple (topic + key) |
| Operational complexity | Lower (single binary) | Higher (ZooKeeper/Kraft, tuning) |
A queue is enough for most background jobs. Kafka is for timelines that many systems need to read.
| Scenario | Use | Why |
|---|---|---|
| Background job: generate PDF report | Queue | One-time task, no replay needed, simple retry is enough |
| Click tracking: 10M events/day, analytics + ML + dashboard | Kafka | Multiple consumers, high throughput, replay for reprocessing |
| Send welcome email after signup | Queue | Simple one-time task, low volume, no history needed |
| Order events: payment, shipping, delivery for audit trail | Kafka | Event sourcing — the timeline is the business record |
| Image thumbnail generation | Queue | One-time processing, serverless-friendly, no replay needed |
| Activity feed: recs + notifications + analytics | Kafka | Three independent consumer groups reading the same events |
Kafka is for timelines many systems need to read. A queue is for tasks one worker should do once. If you can't name why you need replay or multiple consumers — start with a queue.