Buckingham Cloud
Message flow diagram between producers and consumers in a queue

Message queues

Message queues: retries, dead letters, and duplicates

Patterns to keep a failed message from dragging down the rest of the system

When a consumer fails, the queue can become an error amplifier if there is no clear retry policy. We saw this several times in production: a service that responds slowly because of an external dependency starts returning timeouts, the broker requeues the messages, the consumer becomes saturated, and within minutes the queue grows faster than any worker can process. The problem is not the original failure, it is the absence of a plan for when something fails.

This text compares backoff approaches, dead-letter queues, and deduplication by idempotency key, with configuration examples in well-known brokers. The idea is simple: every message that cannot be processed must have a destination decided in advance, not improvised during the incident.

Infinite retries hide design flaws. If a message bounces between the broker and the consumer for hours, no one is looking at why it fails: it is only avoiding acknowledging that this message will not be processed with the current code. An exponential backoff with a bounded ceiling and a maximum number of attempts leaves the problem in plain sight. From there, the decision is binary: the message goes to a dead-letter queue or is discarded with logging.

Measuring the age of pending messages is more useful than looking at the queue size. A queue with ten thousand messages that drain in seconds is healthy; one with two hundred messages that have not moved for an hour is broken. That indicator, together with the rate of messages that end up in the dead-letter queue, makes it possible to detect accumulation before the system goes down.

Idempotency-key deduplication solves the other extreme: retries that do work but generate duplicates. If the consumer processes the same message twice because the ack was lost, the result depends on the business. For non-idempotent operations, it is advisable to store the processed key with a retention window aligned to the maximum retry time. Without that window, deduplication protects nothing.

The closure includes criteria for deciding when to discard a message and when to alert a human. A message with a validation error is discarded and logged. A message that fails due to a downed dependency is retried. A message that has been in the dead queue too long without anyone reviewing it is an alert, not a pending item.

If you want to see how we apply these policies in real services, you can review the services we document or write to us from the contact page. There are also more technical notes in the blog.

Message queues: retries, dead letters, and duplicates

These definitions are not a decorative glossary. They are the points where most arguments arise when a message fails, is retried, and ends up in the dead-letter queue. It is worth putting them in writing before configuring any broker.

Retrying is not the same as insisting

A retry is a decision with a limit: you know how many times you will try, with what spacing, and what to do when it runs out. Insisting is leaving the consumer spinning on the same message with no declared policy. In this text, when we talk about retries we always mean the first definition. Infinite retries are not a strategy: they are a way of hiding that the message will never be processed with the current code, and they usually mask design flaws such as a malformed payload or a failed external dependency that nobody is watching.

What counts as a dead letter

We call a dead letter any message that exhausted its retry policy and fell out of the normal consumption flow. It is not automatic garbage: it is a separate queue where the message still exists, with its original header, the reason for the last failure, and the number of attempts. The distinction matters because a dead letter can be reprocessed after a fix, whereas a message discarded without a record is lost. In the broker configuration examples that appear later, the dead letter always has its own destination, its own retention, and its own alerts.

Idempotency and duplicates

When a consumer processes a message and fails right before acknowledging, the broker may redeliver it. That is not a broker error: it is the expected behavior of at-least-once delivery. Deduplication by idempotency key is what prevents that redelivery from becoming a repeated operation in the database. Important clarification: the idempotency key is not invented by the broker; the producer has to set it and the consumer has to honor it. If the message does not carry a stable key, no deduplication mechanism will work on its own.

Backoff: waiting, not punishment

Backoff is the time waited between a failed attempt and the next one. When we say exponential backoff, we mean that this interval grows with each attempt, not a single formula. It is not a punishment for the message nor a way to silence it: it is a way to avoid hammering a dependency that is already suffering. A backoff without a ceiling can leave a message waiting for hours without anyone noticing, so in the examples there is always an explicit upper limit and a pending-message age metric that makes it possible to see when the queue is piling up.

When to discard and when to alert

Discarding a message is a business decision, not a technical one. If the payload is irrecoverably invalid, retrying it only consumes resources. If the failure depends on an external service that may come back, discarding it loses work. The rule we use in this text is simple: silent discard when the error is deterministic and the message provides no further value; alert a person when the error is transient, when the age of the queue exceeds an agreed threshold, or when the volume of dead letters grows faster than usual. Everything else falls into a gray area and should be reviewed with the team before being automated.

Message queues: retries, dead letters, and duplicates

Buckingham Cloud is not a brand with a single spokesperson. It is a working notebook maintained by a team of platform engineers and SREs, people who spend their days looking at queues, dashboards, and manifests. What gets published here first came out of a real incident, a half-finished migration, or a comparison we ran to decide between two tools.

There are no sponsorships or commissioned content. When a post mentions a broker, a state backend, or a cloud provider, it is because we use it, not because we are paid to name it.

Platform team

Who writes and by what standard

The texts are written by engineers who work on production clusters. Before publishing a comparison, we test it in a staging environment and note down what broke. If a retry pattern works on one broker but not another, we say so.

 Technical review

How we verify what we publish

Every configuration fragment that appears in a post went through a real environment before it made it into the text. The numbers on a cost dashboard or in a queue with pending messages are checked against the screenshot that accompanies them, not estimated by eye.

 Production notes

Where the topics come from

The agenda is set by the problems that show up on call: a consumer that retries endlessly, a Terraform state that gets overwritten between two people, an instance nobody shuts down. We write about it while it is still fresh.

 Readers and colleagues

Who we discuss decisions with

A good part of the corrections and nuances come from other SRE teams and backend developers who read the log. Their comments pushed us to rewrite more than one post about retry policies and dead letter handling.

If you want to follow the thread of what we publish, there is a weekly newsletter and a direct email for technical questions.

Write to the team
Message flow diagram between producers and consumers in a queue

Message queues: retries, dead letters, and duplicates

A failing consumer should not drag down the entire system. These are the pieces worth examining before deciding how many times to retry, where to store what could not be processed, and how to prevent the same message from being executed twice.

Terminal with Kubernetes manifests and resource usage metrics on screen Retries with increasing wait

Exponential backoff and its ceiling

Retrying immediately only amplifies the failure. A wait that grows between attempts gives the dependency time to recover, but it needs a reasonable ceiling: without an upper limit, the last attempt arrives when no one remembers the incident anymore.

Monitoring dashboard with metric charts and event logs Queue age

Measuring the age of the pending message

Queue size is misleading. A small backlog with messages from hours ago indicates a stuck consumer, not a traffic spike. The useful metric is the age of the oldest unprocessed message, not the total count.

Screen with Terraform configuration files and highlighted change plan Dead letter queue

Where what could not be processed ends up

A dead letter queue is not a dumping ground: it is a record of failures that someone must review. If no one looks at it, infinite retries only hide a design problem that is still there.

Containers and services deployed in a cluster with startup logs Idempotency

Keys to avoid duplicating effects

When retrying is unavoidable, the idempotency key prevents the same message from charging twice, sending two emails, or creating two records. The consumer must recognize the key before performing the action, not after.

Chart of cloud service costs with a breakdown by category When to alert

Discard or call a person

Not every failed message deserves human intervention. A validation error is discarded and logged; a sustained failure of an external dependency does justify notifying someone. The difference lies in whether the message can resolve itself.

Message queues: retries, dead letters, and duplicates

Independent technical log, without sponsorship

This entry about message queues is signed by the team of engineers who maintain the log. We write about what we see in production: retries that fire out of control, messages that get stuck, and consumers that fail in a chain. There is no external editorial team behind it and no sponsor setting the tone. When we compare brokers or backoff policies, we do it with configuration snippets and real dashboards on the table, and we also note what did not work. If any of this text helps you make better decisions in your distributed system, it has already served its purpose.

The topics we cover range from Kubernetes and Terraform to observability, instance costs, and legacy service migration. Each article is reviewed before publication and corrected when practice contradicts what we had written.

Cookie settings

We use cookies to keep the site stable, remember basic options, and understand which pages are useful. You can accept, reject, or review the settings before continuing.

ES EN