Retry, Backoff & Dead-Letter Queues

The problem: not all failures are equal

A consumer can fail for two very different reasons:

• Transient: DB connection blip, optimistic-concurrency conflict, broker hiccup. Retrying after a moment usually succeeds.
• Poison: malformed payload, contract violation, a bug. Retrying will fail forever — and an unhandled poison message can block a queue and burn CPU in an infinite redelivery loop.

The failure policy must retry the first kind a bounded number of times, with growing delays, and quarantine the second kind where a human can inspect and replay it.

Our two implementations

Account service (.NET) — in-process retry, error queue

Program.cs → UseMessageRetry(r => r.Intervals(1s, 5s, 25s)): MassTransit catches the exception and re-runs the consumer in-process with delays (exponential-ish, per PLAN.md). After all attempts fail, the message moves to <queue>_error (e.g. account.transfer-requested_error) with the exception details in headers.

Note: domain failures (insufficient funds) never enter this pipeline — they're published as events, not thrown (saga-choreography.md).

Transfer service (NestJS) — broker-level retry topology

No library does it for us here, so the topology is explicit (topology.setup.ts, retry.error-handler.ts):

banking.events ──► transfer.account-events ──handler error──► nack
                        ▲                                       │ (dead-letter)
                        │ TTL expires (5s)                      ▼
                        └─────────────── transfer.account-events.retry
                                  (x-death count grows each lap)

after 3 attempts: publish to banking.dlx ──► banking.dead-letters (parking lot)

• The consumer queue's deadLetterExchange sends nacked messages to the retry queue.
• The retry queue has a 5 s messageTtl and dead-letters expired messages back to the main queue (via the default exchange).
• RabbitMQ's x-death header counts the laps — a persistent, broker-side attempt counter that survives restarts and needs no extra storage.
• At MAX_ATTEMPTS, the error handler publishes the message (plus x-last-error) to the parking lot and acks.

Why not just nack(requeue=true)? Immediate redelivery = a hot loop hammering a failing dependency thousands of times per second. The TTL queue gives delay without blocking the consumer.

The parking lot is an alerting target, not a graveyard

A message in banking.dead-letters means an invariant broke: unknown event type, transfer missing, bug. In production this queue has a length alert (> 0 pages someone), a dashboard, and a replay tool ("shovel" the message back after the fix ships). A DLQ nobody watches is just slow message deletion.

Trade-offs & sharp edges

• Retries amplify load — 3 services each retrying 3× turns one failure into 27 attempts downstream. Bound attempts, use backoff, and (Phase 3) add circuit breakers.
• Per-queue TTL is fixed — true exponential backoff broker-side needs one retry queue per delay tier (5s, 30s, 5m...). We document one tier; adding tiers is mechanical.
• Retries reorder messages — a retried message re-enters behind newer ones. Our defence is the state machine, not ordering assumptions.

Interview Q&A

Q: Walk me through what happens when your consumer throws. Nack without requeue → the queue's DLX routes it to the retry queue → it sits out a 5 s TTL → dead-letters back to the main queue → redelivered with x-death count incremented. Third failure: the error handler publishes it to the parking-lot exchange with the error in a header and acks the original. Total: 3 processing attempts, then quarantine.

Q: Why count attempts with x-death instead of a retry counter in Redis or memory? In-memory counters die with the pod (and are wrong with >1 consumer); Redis adds a dependency and a race. x-death is maintained by the broker itself, exactly-once per dead-lettering, visible in the message for free.

Q: What's the difference between a dead-letter exchange and a dead-letter queue, and your "parking lot"? DLX is RabbitMQ's mechanism (where do rejected/expired messages go); a DLQ/parking lot is the operational concept — the final quarantine humans monitor. We use the DLX mechanism twice: once for retry routing (with TTL) and once for the terminal parking lot. Same primitive, two roles.

Q: How do you replay a parked message safely? Fix the bug first, then re-publish the message to its original routing key (the x-death header preserves it). Idempotency makes replays safe — if part of the work already happened, the inbox and the state machine absorb it. Replay without idempotency is gambling.

Try to break it

1. Publish garbage from the management UI: exchange banking.events, routing key accounts.account.debited, body {"messageId":"poison-1","type":"Accounts.Account.Debited","data":{"transferId":"not-a-real-id"}}. Watch transfer-service logs retry 3× (5 s apart), then find your message in banking.dead-letters with x-last-error explaining why.
2. On the .NET side: publish a transfers.transfer.requested whose data.amountMinor is a string. After retries it lands in account.transfer-requested_error — inspect it in the management UI.
3. Watch a retry lap live: docker compose exec rabbitmq rabbitmqctl list_queues name messages while a poison message bounces between the main and retry queues.