Backend cheatsheetKafka vs RabbitMQPick the right pipe

Kafka vs
RabbitMQ

One keeps a log that readers walk through at their own pace. The other routes each message to the queues that want it. Here is how each works, what each guarantees and how to choose.

Compare the models Go to the decision

By Shree Kumar SharmaKafkaRabbitMQ7 min read

A log or a broker

Kafka keeps an ordered log that many readers walk through at their own pace. RabbitMQ routes each message to queues and deletes it once handled.

ConceptsMust knowLogQueue

Kafka: a logappend(offset 6)Messages stay for the retention period. Each consumer group remembers where it is and can rewind.

RabbitMQ: a brokerroute(order.created.in)Exchanges copy messages into matching queues. An acked message is gone.

	Kafka	RabbitMQ
Unit	Topic split into partitions	Exchange, bindings, queues
After a consumer reads	Message stays; the group moves its offset	Message is removed on ack
Many consumers of one event	Each group reads everything	Bind one queue per consumer type
Ordering	Per partition, by key	Per queue, with one consumer
Who tracks progress	The consumer group (offsets)	The broker (acks)
Sweet spot	Event streams, history, analytics	Tasks, routing, request and reply

Kafka in one picture

Topics, partitions, keys, consumer groups and offsets. Five words that explain almost everything Kafka does.

KafkaPartitionsConsumer groupsOffsets

Picture it: one topic, two consumer groups

Messages with the same key always land in the same partition, which is what keeps one user’s events in order.

Inside a group, each partition is read by one consumer at a time, so a group can never use more consumers than there are partitions. Run this to see how six partitions are shared as a group grows.

assign.jsJavaScript

const partitions = [0, 1, 2, 3, 4, 5];

function rangeAssign(consumers) {
  const per = Math.floor(partitions.length / consumers);
  const extra = partitions.length % consumers;
  let next = 0;
  return Array.from({ length: consumers }, (_, i) => {
    const take = per + (i < extra ? 1 : 0);
    const mine = partitions.slice(next, next + take);
    next += take;
    return 'c' + (i + 1) + '=[' + mine.join(',') + ']';
  }).join(' ');
}

for (const n of [1, 2, 4, 6, 8]) console.log(n + ' consumers: ' + rangeAssign(n));

TerminalOutput

$ node assign.js
1 consumers: c1=[0,1,2,3,4,5]
2 consumers: c1=[0,1,2] c2=[3,4,5]
4 consumers: c1=[0,1] c2=[2,3] c3=[4] c4=[5]
6 consumers: c1=[0] c2=[1] c3=[2] c4=[3] c5=[4] c6=[5]
8 consumers: c1=[0] c2=[1] c3=[2] c4=[3] c5=[4] c6=[5] c7=[] c8=[]

Why it matters: past six consumers the extras sit idle, so choose the partition count for the parallelism you will need.

key'u_91'

Same key, same partition, strict order for that key.

See the example

Ordering

partitions: 6

The ceiling on parallel consumers per group.

See the example

Scale

retention.ms

How long messages stay, read or not. Days, or forever with compaction.

See the example

History

lag

Latest offset minus committed offset. The one Kafka metric to alert on.

See the example

Health

RabbitMQ in one picture

Publishers send to exchanges, exchanges copy into queues by binding rules, and consumers ack each message they finish.

RabbitMQExchangesBindingsAcks

Picture it: one message, three bindings

Exchange type	Routes by	Use for
direct	Exact routing key	Work queues per task type
topic	Key patterns with * and #	Events by region, type or tenant
fanout	Nothing, copies to every bound queue	Broadcasts, cache invalidation
headers	Message header values	Rare; when keys are not enough

In a topic binding, * matches exactly one word and # matches zero or more. Run this to check a few keys against the bindings above.

topic-match.jsJavaScript

function matches(pattern, key) {
  const p = pattern.split('.'), k = key.split('.');
  const walk = (i, j) => {
    if (i === p.length) return j === k.length;
    if (p[i] === '#') return walk(i + 1, j) || (j < k.length && walk(i, j + 1));
    return j < k.length && (p[i] === '*' || p[i] === k[j]) && walk(i + 1, j + 1);
  };
  return walk(0, 0);
}

const bindings = { billing: 'order.*.in', audit: '#', shipping: 'order.shipped.*' };
for (const key of ['order.created.in', 'order.shipped.in', 'order.created.us.east']) {
  const to = Object.keys(bindings).filter((q) => matches(bindings[q], key));
  console.log(key.padEnd(22) + ' -> ' + to.join(', '));
}

TerminalOutput

$ node topic-match.js
order.created.in       -> billing, audit
order.shipped.in       -> billing, audit, shipping
order.created.us.east  -> audit

Why it matters: routing rules live in the broker, so adding a consumer is a new binding, not a code change in the publisher.

Delivery and ordering

Both can lose nothing and both will sometimes deliver twice. Design consumers to be idempotent either way.

ReliabilityAt least onceIdempotencyOrdering

Guarantee	Kafka	RabbitMQ
Publisher knows it landed	`acks=all` with enough in sync replicas	Publisher confirms
Survives a broker restart	Replicated partitions	Durable quorum queues and persistent messages
At least once to consumers	Commit offsets after processing	Manual ack after processing
Duplicates from producer retries	Idempotent producer removes them	Possible; dedupe in the consumer
Exactly once	Transactions for read, process, write inside Kafka	Not offered; use idempotent consumers
Order kept	Within a partition	Within a queue, with one consumer and no requeue

Receivemessage plus its id
Seen before?check a processed ids table
Do the workin one DB transaction
Record the idsame transaction
Ack or commitonly now

The code, side by side

A producer and a consumer for each, with the settings that make them safe by default.

Node.jsKafkaJSamqplibAcks

kafka.tsTypeScript

const kafka = new Kafka({ clientId: 'bridgeq', brokers: process.env.KAFKA_BROKERS!.split(',') });

const producer = kafka.producer({ idempotent: true });
await producer.connect();
await producer.send({ topic: 'messages.sent', acks: -1, messages: [{ key: tenantId, value: JSON.stringify(event) }] });

const consumer = kafka.consumer({ groupId: 'analytics' });
await consumer.subscribe({ topic: 'messages.sent', fromBeginning: false });
await consumer.run({
  eachMessage: async ({ partition, message }) => {
    await analytics.record(JSON.parse(message.value!.toString())); // offset commits after this resolves
  },
});

Why it matters: keying by tenant keeps each tenant’s events in order, and idempotent sends stop retry duplicates. For new projects also look at Confluent’s JavaScript client, which offers a KafkaJS compatible API.

rabbit.tsTypeScript

const conn = await amqp.connect(process.env.AMQP_URL!);
const ch = await conn.createConfirmChannel();
await ch.assertExchange('messages', 'topic', { durable: true });
await ch.assertQueue('whatsapp.send', { durable: true, arguments: { 'x-queue-type': 'quorum', 'x-dead-letter-exchange': 'messages.dlx' } });
await ch.bindQueue('whatsapp.send', 'messages', 'send.whatsapp.*');

ch.publish('messages', 'send.whatsapp.in', Buffer.from(JSON.stringify(msg)), { persistent: true, messageId: msg.id });
await ch.waitForConfirms();

await ch.prefetch(20); // at most 20 unacked messages per consumer
await ch.consume('whatsapp.send', async (m) => {
  try { await whatsapp.send(JSON.parse(m!.content.toString())); ch.ack(m!); }
  catch { ch.nack(m!, false, false); } // to the dead letter exchange, not back to the queue
});

Why it matters: confirms prove the broker has it, prefetch keeps one slow consumer from hoarding, and failures park in a dead letter queue instead of looping.

Choose, then combine

Start from what the consumers need. Many systems end up with both: Kafka for the event stream, RabbitMQ for routing work.

DecisionMust knowTrade offsBoth

Picture it: three questions

BridgeQ eventsmessage.sentdeliveredread kept in Kafka, replayed by analytics

BridgeQ routingsend.whatsapp.*send.sms.*send.email.* routed by RabbitMQ to channel workers

If you need	Lean to
Replay, audit trails, event sourcing	Kafka
Many independent readers of one stream	Kafka
Very high sustained throughput	Kafka
Routing by pattern, per task queues	RabbitMQ
Per message acks, retries and dead letters	RabbitMQ
Priorities, TTL per message, request and reply	RabbitMQ
Simple jobs inside one Node.js service	Neither: BullMQ on Redis

Which one do I need?

Name the job, then pick the tool. When the answer is both, give each the half it is good at.

I want to	Use	Because
Keep every event and replay it later	Kafka	Messages stay for the retention period
Let five teams read the same stream	Kafka	Each consumer group has its own offsets
Route tasks by type or tenant	RabbitMQ	Topic exchanges and bindings do it in the broker
Retry one failed message, then park it	RabbitMQ	Per message nack and dead letter exchanges
Keep one user’s events in order	Kafka	Same key, same partition
Process each message exactly once	Either, with idempotent consumers	Duplicates happen on retries
Queue background jobs in one app	BullMQ	Less to run than either broker