Kafka is the right choice when the log is the product: high throughput, replayable history, multiple consumers reading independently. RabbitMQ wins when the work is the product: complex routing, per-message acks, low-throughput priority queues.
These tools solve different problems even though both move messages. Kafka is a distributed commit log: producers append, consumers read at their own offset, the broker keeps everything for as long as configured. RabbitMQ is a queue server: producers enqueue, consumers dequeue, the broker keeps a message only until someone acks it. Picking the wrong shape forces ugly workarounds either way.
A distributed, append-only commit log. Topics are split into partitions, each an ordered immutable sequence. Consumers track their own offsets. The broker is a storage system more than a router.
A broker with queues fed by exchanges. Producers publish to exchanges, exchanges route to queues by binding rules, consumers pop from queues. Messages disappear on ack. The broker does the routing for you.
Designed for millions of messages per second per cluster. Sequential disk writes, zero-copy, batch compression. LinkedIn famously runs trillions per day.
Tens of thousands of messages per second per node. Beyond that, performance degrades — RabbitMQ Streams (a newer feature) closes the gap but is not the default.
Messages are retained for a configurable time or size, independent of consumption. A new consumer can read from offset 0 and rebuild state. The log is the system of record.
Messages live in the queue until acked. Once acked, they are gone. Replay requires a separate "store" or DLQ pattern. Streams (since 3.9) add log-style retention as an opt-in.
Partition by key. That is the routing. Anything more complex (filtering, fan-out by header, content-based routing) is application code or a stream-processing layer like Kafka Streams or Flink.
Exchanges (direct, topic, fanout, headers) handle the dispatch. Bindings act as routing rules. You can route to dozens of queues from one publish, by pattern.
Strict ordering within a partition. Across partitions, no order. Choose your partition key with that in mind.
Strict FIFO per queue. Single-consumer ordered. Multi-consumer parallelism breaks order unless you set up consistent hashing.
At-least-once by default; exactly-once with transactional producer + read_committed isolation. Mature outbox patterns built around it.
At-least-once with manual acks and publisher confirms. No transactional commit across consumers; "exactly-once" requires application-level idempotency.
High. Requires ZooKeeper (or KRaft mode since 3.5), JVM tuning, partition planning, rebalancing strategies. Confluent Cloud and AWS MSK exist to hide this.
Lower. Single Erlang binary, web management UI, clustering with mirrored queues or Quorum Queues. CloudAMQP and AWS MQ for managed.
Kafka Streams, ksqlDB, Flink, Spark Structured Streaming, Materialize, Debezium for CDC. The richest set of stream-processing tools in software.
Thinner. Some integrations exist via Spring Cloud Stream and connectors, but stream processing is not the goal of the platform.
OpenMessaging Benchmark Framework (LinkedIn), c5n.4xlarge per broker, 1 KB messages, replication factor 3, durable producers. Numbers from the canonical 2020 study (still reproduced regularly).
Source: OpenMessaging Benchmark Framework (Confluent / LinkedIn) ↗
Throughput and latency are not one number each. They depend on workload shape, durability settings, and where the queue sits in your system.
The shape of the workload decides who looks fast. Kafka’s throughput comes from sequential appends, producer batching, and zero-copy reads — a continuous firehose of small messages is its best case, and it degrades gracefully as load climbs. RabbitMQ is the opposite profile: with short queues and moderate load its tail latency is excellent, often beating Kafka at p99, because messages pass through memory and rarely touch disk. Let queues grow deep, though, and RabbitMQ starts paging messages out, throughput drops, and latency goes from the best in the field to the worst. Kafka’s log does not care how far behind a consumer is.
The OpenMessaging benchmark cited above deserves a caveat: it measured sustained throughput and latency on three-broker clusters pushing 1 KB messages with replication factor 3 and durable producers. That is Kafka’s home game — a firehose with no routing. It did not measure topic-exchange fan-out, priority queues, per-message TTLs, or selective routing, which is the work RabbitMQ exists to do. A benchmark of those patterns would flatter RabbitMQ the way the firehose flatters Kafka.
Durability settings dominate the spread more than the engines do. A Kafka producer with acks=1 is answering a different question than one with acks=all and min.insync.replicas=2 — and Kafka famously trusts the OS page cache rather than fsyncing every message, while RabbitMQ’s quorum queues fsync before confirming. Comparing a loosely configured system against a strictly configured one measures the settings, not the software. Most published head-to-heads, including vendor ones, blur this line somewhere.
And most teams never reach territory where any of it matters. Below roughly 50K messages per second, either broker is loafing; the bottleneck is almost always the consumer — deserialization, the database write per message, the external API call. If your throughput target fits in that range, pick by shape (replay and fan-out versus routing and acks), not by benchmark deltas you will never get close to.
Event sourcing or change data capture for downstream consumers
Replayable log, transactional producer, Debezium ecosystem. The canonical event-sourcing substrate.
Background job processing — emails, image resize, async work
Per-message acks, priorities, TTLs, dead-letter exchanges. Workers can be added or removed without partition rebalances.
Real-time analytics ingestion at hundreds of thousands of events/sec
Kafka was built for this. Connect downstream to Flink, Materialize, ClickHouse, or a warehouse.
Microservice request/reply with correlation IDs
RabbitMQ's direct-reply queues handle this cleanly. Kafka can do it but the pattern is awkward.
You're a small team and operational simplicity wins
One binary, web UI, clear primitives. Kafka's ops surface is non-trivial even with KRaft.
You need both shapes — log AND queue — at scale
Both speak the streaming model with simpler ops than Kafka and more durability than RabbitMQ.