CoDel queues
Version 1 used three queues: read, write and slow. CoDel sheds by how long a request waits, not by queue length. Queues use FIFO normally and switch to LIFO under pressure, so fresh requests get served.
Docstore and Schemaless are Uber's MySQL-based distributed databases with thousands of clusters and tens of petabytes. Overload spreads fast because timeouts cause retries, which add load. Many tenants share the fleet, so one tenant must not use all resources.
A stateless query engine plans, routes and authorizes requests. A stateful storage engine runs transactions, pools connections and replicates data. Each partition has one leader and two followers using Raft, on MySQL with local NVMe disks.
Load management sits in the storage engine, where the state lives. Each request enters a priority-aware queue, passes admission checks and is served or rejected early.
Version 1 used three queues: read, write and slow. CoDel sheds by how long a request waits, not by queue length. Queues use FIFO normally and switch to LIFO under pressure, so fresh requests get served.
Scorecard is a rule-based check that caps concurrency per tenant. Regulators watch write bytes, hot partition keys, free memory and goroutine count. They rarely fire and act as a last guard on each node.
Cinnamon ranks requests into tiers t0 to t5, with t1 for user-facing traffic. Missing priorities come from a default per calling service. Queue timeouts follow P90 latency, an auto tuner sets inflight limits, and PID control sheds only when needed.
External signals, such as follower commit lag, feed the same control loop as local signals. A framework called BYOS routes each signal to broad priority shedding or to caller-specific shedding.
| No. | Chose | Instead of | Why |
|---|---|---|---|
| D1 | Use concurrency as the load signal | QPS limits | Concurrency follows Little's Law and tracks real resource use better than request counts. |
| D2 | Shed at the storage node | Quotas in the query layer with a central Redis | Redis added a hop and a failure point, and routers lacked partition health. |
| D3 | Route remote signals into the local loop | External token-bucket limiters | They caused split-brain behavior and poor global decisions. |
| CoDel | Controlled Delay, a queue policy that drops work based on wait time. |
|---|---|
| PID control | A feedback method that adjusts output from current error, past error and trend. |
| Little's Law | Concurrency equals throughput times latency. |