Anurag’s Read

Worker Backpressure (Part 1).

Shows how a feedback loop in a shared queue library kept two incidents from flooding the dead letter queue. It has failure counts and honest limits.
Score
8.7
Verdict
Must read
Blog and date
Canva,
Original length and level
9 min, intermediate
Topics
reliability, queues, backpressure, incident resilience
Read the original

Context

Canva queue workers do asynchronous jobs such as asset resizing, email, and export rendering. A worker pulls messages whenever it has spare capacity, which gives low latency when all is healthy. When a dependency fails, the worker keeps pulling and slows the recovery of that dependency.

Doomed messages retry and then reach the dead letter queue (DLQ), where staff drain them by hand. These failures also page the on-call engineers.

Architecture

Worker Backpressure lives in the shared queue library and runs fully inside each worker. After each message, the worker records success or failure. A pluggable Backpressure Controller reads these outcomes and keeps one value, the backoff factor.

The factor ranges from 0.0 for full speed to 1.0 for full back off. Before each poll, the worker asks how many messages it may process at the same time. The controller scales that number down by the backoff factor. Unpulled messages stay in the queue.

Mechanisms

M1

Backoff factor

The controller updates the factor after every success or failure. The update costs one arithmetic operation. The factor rises when failures exceed the set point and falls when they stay below it.

M2

Set point

The set point is the failure rate the team accepts as background noise. Below it, the controller does nothing. Both pilot workers used 5%. Each instance judges only its own outcomes, so one bad instance can back off while the fleet average stays low.

M3

Pluggable controller interface

The controller is an interface, so teams can swap the algorithm. The design follows the adaptive throttling idea in the Google SRE chapter on handling overload.

Design decisions

No.ChoseInstead ofWhy
D1Throttle inside the shared queue library at the worker levelManual scaling, fixed rate limits, circuit breakers, and exponential backoffExtra workers add load to a failing dependency, and fixed limits assume static capacity. Circuit breakers have no gradual ramp, and backoff only smooths each message.
D2Use only success and failure as the health signalLatency or messages in flightIt is the simplest signal and needs no extra data. The team may add other signals later.

Key numbers

  • The first incident lasted about 4 hours with intermittent error spikes.
  • In that incident the worker handled 1,610,173 successes and 498 errors.
  • The fleet failure peak was 1.42%, and single instances reached about 50%.
  • The second incident lasted about 32.5 hours and the fleet peak was 3.7%.
  • Only 22 messages reached the DLQ after 5 failed attempts each.
  • Without backpressure the author estimates at least 16,000 DLQ messages.

Failures and limits

  • Backpressure lowers errors by lowering throughput, and a worker at full capacity would feel this more.
  • One signal may not suit every workload.
  • The 16,000 message estimate assumes independent failures, which the author admits is only a lower bound.
  • Part 1 does not explain how a fully backed off worker detects recovery. Part 2 will cover it.

Lessons you can use

  • A queue can only push back by leaving messages unpulled, so build this into the shared library.
  • Judge health per instance, so one sick node slows itself without slowing the fleet.
  • Start with a minimal control loop and measure it in real incidents.

Glossary

Dead letter queueA queue that holds messages which failed too many delivery attempts.
Set pointThe target value a feedback controller tries to hold, here the accepted failure rate.
BackpressureA way for a consumer to slow its input when it cannot keep up.