Anurag’s Read

How Uber Protects Against Retry Storms.

Explains retry amplification with clear math and a practical fix that lets only the service that owns an error retry. It covers trade-offs, a real incident, and measured results that transfer to any microservice system.
Score
8.9
Verdict
Must read
Blog and date
Uber,
Original length and level
9 min, advanced
Topics
reliability, microservices, retries, incident learnings
Read the original

Context

Uber teams tune retries and retry budgets by hand for each service. This gives no view of amplification across deep call chains. Retries ignore context, so teams control how many retries happen but not when.

Take a seven-node chain with one retry per hop. A failure at node D makes nodes D through G serve 8 times the load.

Architecture

Each error gets an owner, and only the edge where the error began retries. Retry middleware (RetryMW) runs on the caller side. The Service Dependency Analysis (SDA) system correlates inbound failures with outbound failures.

A service whose outbound calls did not fail claims its error. The claim travels to the caller in the x-uber-error-claim header. The caller reads it to decide whether to retry and passes the claim state upstream.

Mechanisms

M1

Claim rules

If the callee did not fail and the caller failed, the caller claims the error. If both failed and the claim is claimed or missing, the caller retries and then unclaims. If the claim is unclaimed, the caller does not retry and passes the unclaim upstream. A missing header marks an uncooperative environment, so the first node to see it unclaims the error.

M2

Coincidental errors

SDA blames downstream failures first, so a service's own errors can be unclaimed by mistake. In the example, about 1.9 of 10 internal errors are wrongly unclaimed. Six months of data put the worst case near 2%. SDA now unclaims only when inbound failures correlate with outbound failures on fail-close dependencies.

M3

At-least-once retries

Many services have no retries on fail-close calls, so removing caller retries would cut availability. RetryMW sets a retry-criteria-satisfied flag for SDA. No new retries are added, and the guarantee needs one retrying service in the chain.

M4

Context drops

If context breaks inside a service, SDA cannot link calls, so that service claims the error. This allows extra retries but still stops the unclaimed error at the next hop.

Design decisions

No.ChoseInstead ofWhy
D1An ownership headerTranslating downstream error codes upstreamIt does not scale with fan-in, fan-out, and changing call flows.
D2Retries on the origin edge onlyA retry budget at every hopBudgets still compound as (1+B) to the power of depth.
D3Unclaim only on correlated failuresUnclaim on any downstream failureIt misattributes coincidental errors.

Key numbers

  • The maximum retry storm radius fell from 25 to 3.
  • The average radius fell from 20 to 2.
  • A 10% budget at every hop gives 1.33 times the load.
  • On November 18, 2025, callers avoided up to 200,000 extra requests.
  • About 9.5 million spurious requests were stopped across ancestors.
  • Simple budgets would have raised traffic by 46% to 135%.
  • The worst-case coincidence rate is about 2%.

Failures and limits

  • The availability model assumes independent errors and recovery by retry, and overload breaks both.
  • Untracked dependencies are a blind spot.
  • A missing header gives partial protection.
  • The post gives no latency overhead.

Lessons you can use

  • Decide which service owns an error and let only that edge retry.
  • Pass ownership in a header.
  • Build it in shared middleware.
  • Treat a missing signal as a safe default that limits retries.

Glossary

Retry stormA surge of retries that multiplies load on a service that already fails.
Retry budgetA cap on retries as a share of normal requests.
Fail-close dependencyA dependency whose failure makes the caller fail.