Anurag’s Read

How Uber Conquered Database Overload: The Journey from Static Rate-Limiting to Intelligent Load Management.

Shows three generations of overload control on a large database fleet, with reasons each failed and before/after numbers. Product engineers can reuse the ideas of priority tiers, adaptive limits and shedding near the state.
Score
8.8
Verdict
Must read
Blog and date
Uber,
Original length and level
14 min, advanced
Topics
load shedding, databases, reliability, rate limiting
Read the original

Context

Docstore and Schemaless are Uber's MySQL-based distributed databases with thousands of clusters and tens of petabytes. Overload spreads fast because timeouts cause retries, which add load. Many tenants share the fleet, so one tenant must not use all resources.

Architecture

A stateless query engine plans, routes and authorizes requests. A stateful storage engine runs transactions, pools connections and replicates data. Each partition has one leader and two followers using Raft, on MySQL with local NVMe disks.

Load management sits in the storage engine, where the state lives. Each request enters a priority-aware queue, passes admission checks and is served or rejected early.

Mechanisms

M1

CoDel queues

Version 1 used three queues: read, write and slow. CoDel sheds by how long a request waits, not by queue length. Queues use FIFO normally and switch to LIFO under pressure, so fresh requests get served.

M2

Scorecard and regulators

Scorecard is a rule-based check that caps concurrency per tenant. Regulators watch write bytes, hot partition keys, free memory and goroutine count. They rarely fire and act as a last guard on each node.

M3

Cinnamon

Cinnamon ranks requests into tiers t0 to t5, with t1 for user-facing traffic. Missing priorities come from a default per calling service. Queue timeouts follow P90 latency, an auto tuner sets inflight limits, and PID control sheds only when needed.

M4

Unified engine

External signals, such as follower commit lag, feed the same control loop as local signals. A framework called BYOS routes each signal to broad priority shedding or to caller-specific shedding.

Design decisions

No.ChoseInstead ofWhy
D1Use concurrency as the load signalQPS limitsConcurrency follows Little's Law and tracks real resource use better than request counts.
D2Shed at the storage nodeQuotas in the query layer with a central RedisRedis added a hop and a failure point, and routers lacked partition health.
D3Route remote signals into the local loopExternal token-bucket limitersThey caused split-brain behavior and poor global decisions.

Key numbers

  • Tens of millions of requests per second
  • Throughput under overload up 80% (5,400 vs 3,000 QPS)
  • p99 latency down about 70%
  • Goroutines down about 93% (10,000 vs 150,000 peak)
  • Heap down about 60% (1 GB vs 5-6 GB)
  • Six priority tiers, t0 to t5

Failures and limits

  • CoDel treated all requests the same, so user traffic dropped with background work.
  • Fixed timeouts and limits needed constant manual tuning and caused retry herds.
  • The post gives no thresholds, PID gains or test conditions.

Lessons you can use

  • Protect user-facing traffic first with explicit priorities.
  • Reject early to save work and avoid memory exhaustion.
  • Replace static limits with feedback control.
  • Put control close to the resource being protected.

Glossary

CoDelControlled Delay, a queue policy that drops work based on wait time.
PID controlA feedback method that adjusts output from current error, past error and trend.
Little's LawConcurrency equals throughput times latency.