Backoff factor
The controller updates the factor after every success or failure. The update costs one arithmetic operation. The factor rises when failures exceed the set point and falls when they stay below it.
Canva queue workers do asynchronous jobs such as asset resizing, email, and export rendering. A worker pulls messages whenever it has spare capacity, which gives low latency when all is healthy. When a dependency fails, the worker keeps pulling and slows the recovery of that dependency.
Doomed messages retry and then reach the dead letter queue (DLQ), where staff drain them by hand. These failures also page the on-call engineers.
Worker Backpressure lives in the shared queue library and runs fully inside each worker. After each message, the worker records success or failure. A pluggable Backpressure Controller reads these outcomes and keeps one value, the backoff factor.
The factor ranges from 0.0 for full speed to 1.0 for full back off. Before each poll, the worker asks how many messages it may process at the same time. The controller scales that number down by the backoff factor. Unpulled messages stay in the queue.
The controller updates the factor after every success or failure. The update costs one arithmetic operation. The factor rises when failures exceed the set point and falls when they stay below it.
The set point is the failure rate the team accepts as background noise. Below it, the controller does nothing. Both pilot workers used 5%. Each instance judges only its own outcomes, so one bad instance can back off while the fleet average stays low.
The controller is an interface, so teams can swap the algorithm. The design follows the adaptive throttling idea in the Google SRE chapter on handling overload.
| No. | Chose | Instead of | Why |
|---|---|---|---|
| D1 | Throttle inside the shared queue library at the worker level | Manual scaling, fixed rate limits, circuit breakers, and exponential backoff | Extra workers add load to a failing dependency, and fixed limits assume static capacity. Circuit breakers have no gradual ramp, and backoff only smooths each message. |
| D2 | Use only success and failure as the health signal | Latency or messages in flight | It is the simplest signal and needs no extra data. The team may add other signals later. |
| Dead letter queue | A queue that holds messages which failed too many delivery attempts. |
|---|---|
| Set point | The target value a feedback controller tries to hold, here the accepted failure rate. |
| Backpressure | A way for a consumer to slow its input when it cannot keep up. |