Anurag’s Read

ZGateway: Learnings from Putting a Proxy in Front of ZippyDB.

Shows when a proxy tier pays for itself in front of a huge key-value store, with a fan-in model, load shedding results and rollout controls. Useful for anyone running shared infrastructure with many clients.
Score
8.6
Verdict
Must read
Blog and date
Meta,
Original length and level
15 min, advanced
Topics
proxy, key-value store, load shedding, reliability
Read the original

Context

ZippyDB serves hundreds of teams from over a million client hosts that Meta cannot update quickly. Each client opened connections to every database host it needed, a dense many-to-many TLS mesh. The mesh used memory, CPU and file descriptors even when idle, and each new client cohort hurt every database host.

Reconnection storms were the main risk, and one routing bug exhausted file descriptors across the fleet.

Architecture

ZGateway is a stateless proxy tier in each region, found through ServiceRouter. A client keeps a sticky connection to a regional gateway host. The host ends TLS, checks use-case access lists, and applies per-tenant admission control.

It resolves the target shard and checks a local cache on caching tiers. Misses and writes are batched with other in-flight requests for the same shard and sent to replicas. Responses return to callers, and metrics and quota use are recorded per use case.

Key-to-shard mapping and replica hedging stay in the embedded C++ client, so the gateway handles traffic management only.

Mechanisms

M1

Fan-in model

Meta modeled shards as balls thrown into database hosts. The chance that a host is hit is one minus e to the power of minus shards over hosts. With mock inputs of 500,000 database hosts and 1,000,000 clients, per-host connections fell about 97 to 98%. Database fan-in no longer grows with client count.

M2

Batching and coalescing

A shared batcher groups requests by use case and physical shard into one backend call. Concurrent requests for one key trigger a single fetch whose result goes to all callers. A batch flushes on a time window, a size limit or a request count cap. Idle batch entries expire, and a cap on in-flight batches rejects new work when the backend is slow.

M3

Discriminant load shedding

Each request maps to a bucket for its use case and priority, and buckets drain round-robin. A flooding tenant fills only its own bucket and gets shed. A CPU controller adjusts a shared token bucket with additive increase and multiplicative decrease. In a test above 90% CPU, only 6 of about 1,350 buckets shed requests.

M4

Weighted load balancing

A control loop reads recent CPU of each host and lowers the weight of busy hosts. Hosts range from about 26 to 126 cores, so new hosts start with weight scaled to capacity. Changes are damped, clamped and limited to the most imbalanced hosts to avoid costly shard moves.

M5

Cache with live invalidation

Hot reads come from an in-process cache, and a per-key lock turns a thundering herd into one fetch. A change stream of writes and checkpoints invalidates or refills entries within a stated staleness bound. Each host owns a key slice through consistent hashing.

Design decisions

No.ChoseInstead ofWhy
D1Add a proxy tier between clients and database hostsKeep direct client accessFan-in grew with adoption and became a reliability limit. Meta accepted an extra hop and a few percent of CPU.
D2Move batching into the gatewayPer-client batching librariesClient libraries were fragile, CPU heavy and could merge only inside one process.
D3Use an adaptive balancer policyOne fixed balancing policyOne policy could not serve both a calm tier and a stressed tier.
D4Allow cross-region failover with mega-regions and ringsFailover inside one region onlyAn overloaded region queued requests while nearby capacity sat idle.

Key numbers

  • Over 1 billion operations per second pass through ZGateway.
  • About 40% of ZippyDB traffic uses it, projected above 60%.
  • Proxy overhead is about 6% compute for an average use case.
  • Total persistent connections fall about 19x in the model.
  • Load shedding machinery costs about 8% CPU, with goodput near 97 to 98%.
  • In the overload test, 1,344 of 1,350 buckets served 99.9% of requests.

Failures and limits

  • A routing bug made every client open a connection per shard, and hosts crashed from file descriptor exhaustion in a reboot loop.
  • Hot gateway hosts caused error spikes and ServiceRouter throttling.
  • Batching holds requests in memory, so out-of-memory risk is reduced but not removed.
  • Cache reads are only bounded-stale, and one process hosts many tenants, so one tenant can threaten the host.

Lessons you can use

  • Direct client access works at small scale, but plan a proxy before client fan-in becomes the reliability limit.
  • Isolate tenants with per-tenant buckets so one noisy caller cannot take down a shared tier.
  • Roll out with percentage knobs, region filters and a kill switch so no client code changes are needed.
  • Model connection counts before building to estimate the benefit.

Glossary

ZippyDBMeta's most widely used key-value store.
ServiceRouterMeta's service discovery and routing layer.
Fan-inThe number of inbound connections one server receives.
CoalescingMerging concurrent requests for the same key into one backend fetch.