Anurag’s Read

Avoiding insurmountable queue backlogs.

Shows how queues that add availability can make recovery much slower, and gives concrete designs such as shuffle-sharded queues, sidelining and heartbeats. Strong material for anyone who runs async workers.
Score
8.7
Verdict
Must read
Blog and date
AWS,
Original length and level
19 min, advanced
Topics
queues, reliability, multi-tenancy, backpressure
Read the original

Context

Services such as AWS Lambda, AWS IoT Core and SQS use queues to store work until a consumer is ready. Queues improve durability and availability because they retry failed work. The cost is latency, and a large backlog can take a very long time to clear.

Multi-tenant services add a second problem, because one customer can fill a shared queue and hurt others.

Architecture

A producer sends a message to a queue through a thin API that authenticates the caller and tags the message with caller data. Consumers poll the queue, process each message and then delete it. If a consumer fails, the message becomes visible again after the visibility timeout.

Lambda asynchronous invocations use this model and keep a durable queue for retries. For multi-tenant systems, Amazon uses a fixed set of queues and maps each customer to a few of them. Consumers also move late or excess messages to side queues, so fresh work runs first.

Mechanisms

M1

Shuffle-sharded queues

Lambda keeps a fixed set of queues. Each customer hashes to a small subset of them. When a customer enqueues work, the system picks the least full queue in that subset. A noisy customer then fills only its own queues.

Other customers share few or no queues with that customer.

M2

Sidelining excess and old traffic

A consumer checks the rate of each customer. Messages over that rate move to a spillover queue, which approximates a priority queue. Messages older than an age threshold move to a backlog queue that runs after the live queue. This approximates last in, first out processing, so real-time data runs first.

Systems that resync data on a schedule can drop messages older than the last full sweep.

M3

Concurrency limits from Little's Law

Concurrency equals arrival rate times latency. At 100 messages per second and 100 ms latency, a worker needs about 10 threads. At 10 s latency it needs about 1,000 threads. One slow workload can use all threads, so teams use non-blocking I/O, per-workload semaphores and thread pools.

Auto scaling limits also help.

M4

Heartbeating long messages

If processing takes longer than the visibility timeout, SQS delivers the message again. Parallel copies then multiply work and can cause a fork-bomb brownout. The consumer can stop work at expiry. It can also call the API to extend the visibility timeout, much like renewing a lease.

M5

Age based metrics

Producer error rates show only queue availability. Consumer error rates look too high because retries often succeed. Message age, computed from the enqueue timestamp, shows true latency. AWS IoT added AgeOfFirstAttempt to ignore retries and slow readers.

It also added AgeOfFirstSubscriberFirstAttempt for fan-out.

Design decisions

No.ChoseInstead ofWhy
D1Use a fixed set of shuffle-sharded queues.One queue per workload.Polling cost grows with the number of queues, so it cannot scale to millions of devices.
D2Sideline old and excess messages to other queues.Process everything in arrival order.Arrival order lets a backlog of stale work delay new work for hours.
D3Alarm on any message in a dead-letter queue.Rely only on dead-letter volume as an availability signal.Dead-letter data arrives too late to be the only signal.

Key numbers

  • After a one hour outage, clearing the backlog needs double capacity for one more hour.
  • A 30 minute operator response with a backlog 10 times consumer capacity means about 300 minutes to drain.
  • At 100 messages per second, 100 ms latency needs about 10 threads and 10 s needs about 1,000.
  • SQS FIFO polling covers only the most recent 20,000 unprocessed messages.

Failures and limits

  • Backlogs grow when enqueue spikes, processing costs rise or a dependency slows or fails.
  • Bugs that skip message deletion hold messages in flight until the visibility timeout ends.
  • In SQS FIFO queues, a large backlog in some message groups can starve other groups.
  • If polls always return messages, the fleet has no spare capacity for bursts.
  • Shared SQS queues cannot count messages by attribute, so backpressure per customer is hard.

Lessons you can use

  • Plan how long a backlog takes to drain, not only how fast the system runs when healthy.
  • Measure message age and alarm on it.
  • Isolate tenants with separate or sharded queues, and move old or excess work aside.
  • Cap threads per workload and heartbeat long jobs.

Glossary

Shuffle-shardingA method that gives each customer a small random subset of shared resources.
Visibility timeoutThe time a received SQS message stays hidden from other consumers.
Dead-letter queueA queue that holds messages that the system could not process.
Little's LawAverage concurrency equals arrival rate multiplied by average time in the system.