Message Queues and Streams

One-liner: A broker decouples producers from consumers in time and rate; queues deliver each message to one worker, topics fan out to many.

Say this#

SAY THIS

Queue (point-to-point) — each message consumed by exactly one consumer; work distribution (SQS, RabbitMQ queue).
Topic (pub/sub) — each message delivered to every subscriber group (Kafka topic, SNS, RabbitMQ exchange fan-out).
Partition — ordered shard of a topic; unit of parallelism and ordering (Kafka, Kinesis shards, Pulsar).
Consumer group — set of consumers sharing a topic's partitions; each partition → one consumer in the group.
Offset — consumer's position in a partition; committed to track progress.
Ack / visibility timeout — consumer confirms; unacked messages are redelivered (SQS visibility timeout default 30 s).
At-most-once / at-least-once / exactly-once — delivery guarantees; at-least-once + idempotent consumer is the practical target.
Ordering — per-partition / per-key in Kafka; FIFO queues in SQS (per message group).
Dead-letter queue (DLQ) — where poison messages go after N failed attempts.
Backpressure — consumers slow → lag grows (Kafka) or queue depth grows; producers throttle or shed.
Consumer lag — how far behind the head; the metric to alert on.
Retention / log compaction — Kafka keeps messages for time/size or keeps latest per key.
Push vs pull — RabbitMQ pushes to consumers; Kafka consumers pull.
Broker examples — Kafka, RabbitMQ (AMQP), SQS/SNS, Google Pub/Sub, Redis Streams, NATS, Pulsar, ActiveMQ.

How it works#

TEXT

producers -> [topic: orders | p0 | p1 | p2 | p3]  -> consumer group A (4 consumers, 1 partition each)             key=user_id -> hash -> partition     -> consumer group B (analytics, independent offsets)producers -> [topic: orders | p0 | p1 | p2 | p3]  -> consumer group A (4 consumers, 1 partition each)
             key=user_id -> hash -> partition     -> consumer group B (analytics, independent offsets)

Producer sends with a key; broker hashes key → partition (same key ⇒ same partition ⇒ ordered).
Broker persists (Kafka: append-only log replicated across brokers, acks=all for durability).
Consumers in a group pull from their partitions, process, commit offsets.
Failure: uncommitted messages are reprocessed → at-least-once → consumer must be idempotent.
Repeated failure → DLQ after max retries; alert and inspect.

Queue vs stream: a queue deletes on ack (RabbitMQ, SQS); a log retains and lets many groups replay (Kafka, Kinesis).

Trade-offs#

Choice	Pro	Con
Kafka	Huge throughput, replay, ordering per key, many consumer groups	Operational weight (brokers + KRaft/Zookeeper), partitions fixed-ish, no per-message ack
RabbitMQ	Flexible routing (exchanges), per-message ack, low latency, priorities	Lower throughput; no replay; queues are the bottleneck
SQS	Zero ops, infinite scale, cheap	No ordering (standard), 256 KB messages, no fan-out alone (pair with SNS)
Redis Streams	Already have Redis, simple	Memory-bound, weaker durability
More partitions	More parallelism	More files/metadata, rebalance pain, less global ordering
Sync RPC instead	Immediate response, simple	Tight coupling, cascading failures, no buffering

Numbers#

Kafka: millions of msgs/s per cluster; a single partition commonly 10-50 MB/s; end-to-end latency ~5-20 ms.
RabbitMQ: tens of thousands of msgs/s per node order of magnitude.
SQS: standard queues effectively unlimited throughput; FIFO 300 msgs/s per group without batching (3,000 with batching, higher in high-throughput mode); max message 256 KB; retention up to 14 days (default 4).
Kafka default retention 7 days; max message size default 1 MB.

Say this#

How it works#

Trade-offs#

Numbers#

In the interview#