Design reliable message-driven systems using queues and event streaming. Covers queue selection, delivery guarantees, dead letter handling, consumer patterns, and production monitoring.
Message queues decouple producers and consumers of work, enabling systems to scale independently, tolerate failures, and process events asynchronously. This architecture matters when a service must handle more work than it can process in real time, when components need to be resilient to outages, or when data must flow reliably across distributed services. It is the foundation for event-driven backends, background job processing, and real-time data pipelines.
A message queue system consists of three primary elements: a producer that sends messages, a queue that stores messages persistently, and a consumer that retrieves and processes messages. Messages are structured as payloads—typically JSON or binary—sent to a named queue. The producer writes messages using a standardized API; the queue guarantees delivery, ordering, and durability; and the consumer fetches messages from the queue and acknowledges completion.
Messages are sent via a protocol such as AMQP, MQTT, or HTTP. A typical send operation uses a POST request to a queue endpoint:
POST /queues/orders/incoming HTTP/1.1
Host: message-queue.example.com
Content-Type: application/json
{
"order_id": "ORD-1024",
"customer_id": 4821,
"items": [
{ "product_id": 104, "quantity": 3 }
],
"timestamp": "2025-04-05T10:20:30Z"
}The queue stores the message on disk and assigns it a unique identifier. Consumers poll or subscribe to the queue, retrieving messages in a first-in, first-out (FIFO) order. Each message can be acknowledged with an ACK after processing, which removes it from the queue. If a consumer crashes before acknowledging, the message is redelivered to another consumer or the same one.
When a consumer fetches a message, it receives it in a receive phase. The message is marked as "in-flight" and is not available to other consumers during this time. If the consumer fails before ACKing, the message is requeued after a timeout. The queue system supports configurable delivery semantics:
To implement exactly-once processing, the consumer includes a message_id in its ACK and the queue maintains a deduplication store. If a message_id is received again, the queue skips reprocessing. The ACK includes a delivery_tag and a timestamp, and the queue persists the acknowledgment.
Failure modes are common and subtle:
ACK before crashing. The message is reprocessed, leading to duplicates.ACK. The message becomes available again and is picked up by another consumer, causing duplication.Queues are configured with retention policies. Messages can be retained for a fixed time (time_to_live) or until a certain number of messages are enqueued (max_messages). A queue can be set to expire after a period of inactivity.
In configuration, the key max_length defines the maximum number of messages stored. When the queue reaches this limit, new messages are discarded or moved to a dead-letter queue. The max_delivery_attempts key controls how many times a message can be retried before being moved to a dead-letter queue. Each retry may include exponential backoff, where the delay between retries grows over time.
A common pattern is to route failed messages to a dead-letter queue (DLQ). Messages are moved to the DLQ when they exceed max_delivery_attempts. The DLQ is monitored and processed separately. A consumer can be configured to process DLQ messages and send alerts on failure:
queue:
name: "payments.processed"
max_delivery_attempts: 5
time_to_live: "PT1H"
max_length: 10000
dead_letter_queue:
name: "payments.failed"
max_length: 1000
message_ttl: "P7D"The dead-letter queue can be consumed by a dedicated service or a script that logs all failures and sends a summary email.
Consumers are typically deployed in a cluster and share a queue. Each consumer pulls messages using a receive operation, and the queue balances load by distributing messages across all active consumers. A consumer pulls a batch of messages in one request, improving throughput.
To reduce round trips, the receive operation supports batching:
GET /queues/events/receive?batch_size=10&wait_time=30s HTTP/1.1
Host: message-queue.example.com
Authorization: Bearer <token>
200 OK
Content-Type: application/json
[
{
"message_id": "MSG-12345",
"body": { "event": "user.created", "user_id": 721 },
"delivery_tag": 1001,
"timestamp": "2025-04-05T10:21:00Z"
},
{
"message_id": "MSG-12346",
"body": { "event": "order.placed", "order_id": "ORD-1025" },
"delivery_tag": 1002,
"timestamp": "2025-04-05T10:21:02Z"
}
]The wait_time parameter enables long polling: the server holds the request open until a message arrives or the timeout expires. This reduces latency and network overhead.
When consumers scale dynamically, the queue must handle bursts. A consumer that starts up automatically pulls messages from the queue and begins processing. The queue assigns messages to consumers based on availability and load. If a consumer fails, its in-flight messages are returned to the queue and become available for other consumers.
For systems requiring strict ordering, queues support ordered delivery. A message can be assigned a sequence_number, and the queue ensures that messages with lower sequence numbers are delivered before those with higher numbers. This is essential for event sourcing or state machine processing.
A common mistake is assuming FIFO order across multiple queues. In reality, messages may be reordered if consumers process different batches at different speeds. To enforce ordering, a message may include a group_id, and the queue delivers messages from the same group in sequence.
The ack_mode setting controls how acknowledgments are handled:
auto_ack: The queue acknowledges the message immediately after sending it to the consumer.manual_ack: The consumer must explicitly send ACK after processing.ack_once: The consumer acknowledges the entire batch after processing all messages.The ack_mode can be set per consumer, per queue, or per message. A misconfigured ack_mode can lead to undelivered messages or duplicate processing.
When a consumer cannot keep up with incoming messages, backpressure builds. The queue can be configured with a flow_control setting that limits the number of messages a consumer can process at once. The consumer fetches messages and then sends flow_control updates to the queue:
PUT /queues/events/flow_control HTTP/1.1
Host: message-queue.example.com
Content-Type: application/json
{
"consumer_id": "worker-03",
"max_in_flight": 100,
"credit": 25
}The credit value defines how many messages the consumer can receive before sending another update. The queue sends messages to the consumer until the credit is exhausted, then pauses. This prevents memory overflow and ensures smooth processing.
If a consumer becomes unresponsive, the queue can send a pause command:
POST /queues/events/pause HTTP/1.1
Host: message-queue.example.com
Content-Type: application/json
{
"consumer_id": "worker-03",
"reason": "high_load",
"duration": "PT5M"
}The consumer stops fetching messages for five minutes, allowing the system to catch up.
Messages are often structured as JSON, but schemas are essential for compatibility across services. A common pattern is to include a schema_version field in the message:
{
"schema_version": "1.2",
"event_type": "user.created",
"payload": {
"user_id": 721,
"email": "[email protected]",
"created_at": "2025-04-05T10:20:30Z"
},
"metadata": {
"source": "api-gateway",
"correlation_id": "CORR-8833"
}
}Consumers must be able to handle multiple schema versions. A consumer can be configured to use a specific version or automatically resolve schemas from a registry. If a message has a schema_version not supported by the consumer, the system can trigger a schema migration.
Schema evolution is handled by versioning. When a new field is added, the consumer reads the old schema and fills missing fields with defaults. When a field is removed, the consumer ignores it. A message with schema_version: "1.0" may be processed by a consumer that expects 1.1, and the consumer uses backward compatibility.
A message queue system emits errors that are critical to observe:
message_lost: A message was sent but not acknowledged.consumer_stalled: A consumer has not sent ACK for over max_delivery_interval.queue_full: The queue has reached max_length and is dropping messages.dead_letter_threshold_exceeded: The number of messages in the dead-letter queue exceeds a threshold.These errors are logged and can trigger alerts. A consumer can be configured with a monitoring_endpoint that reports its status:
POST /queues/events/monitor HTTP/1.1
Host: message-queue.example.com
Content-Type: application/json
{
"consumer_id": "worker-03",
"status": "running",
"in_flight_messages": 127,
"last_ack_timestamp": "2025-04-05T10:23:45Z",
"queue_name": "events"
}The queue system maintains a health dashboard showing:
A common production failure is a consumer that processes messages but fails to send ACK due to a network error. The consumer logs the message but never acknowledges it. The queue detects the failure and requeues the message. Over time, the same message is processed multiple times, leading to duplicate events.
To prevent duplication, consumers must process messages idempotently. A consumer uses the message_id to track which messages have been processed. On receiving a message, it checks a database or cache for an entry with that message_id. If found, it skips processing.
If a consumer fails to process a message, it can retry with a configurable strategy:
The queue system tracks retry attempts and includes metadata in the message:
{
"message_id": "MSG-12345",
"attempts": 3,
"last_error": "database.connection.timeout",
"retry_delay": "PT1M",
"next_retry": "2025-04-05T10:35:00Z"
}When a message is retried, the queue updates the attempts counter and schedules the next retry.
A common pattern is event sourcing, where every state change is a message. A service publishes events to a queue, and other services consume them to update views or trigger actions.
Another pattern is job processing: a service creates a job, puts it in a queue, and workers process the job. Workers can be specialized—some handle image processing, others handle email delivery.
A third pattern is fan-out: a single message triggers multiple consumers. A user.created event may be consumed by a user profile service, a notification service, and a billing service. Each consumer processes the event independently.
These patterns rely on message queues as the central nervous system of the backend. The architecture enables systems to scale independently, recover from failures, and evolve over time—without breaking existing components.
This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy