Anurag’s Read

Building a data plane from scratch: Stripe's own high-performance distributed proxy.

A rare account of replacing Envoy with an in-house Go proxy, with clear reasons such as connection counts, risky upgrades, and discovery overhead. The post gives real latency and CPU gains and is honest about the risk of building your own critical component.
Score
8.6
Verdict
Must read
Blog and date
Stripe,
Original length and level
7 min, advanced
Topics
service mesh, networking, reliability, go
Read the original

Context

Stripe aims for 99.9995% reliability. It used Envoy as its service mesh proxy for almost 10 years. Envoy runs one worker per core, and each worker holds its own upstream connections. Shared services got hundreds of thousands of connections. Upgrades also broke unusual services, and routing for large Ruby machines needed heavy Envoy changes.

Architecture

Stripe wrote mesh-proxy in Go. It has its own HTTP/1.1 and HTTP/2 code, built on Go's HTTP/2 framing library. It reads the service registry directly and skips xDS. A client proxy resolves the target hosts and applies priority routing, rate limits, retries, and hedging. It sends requests over one shared connection to each upstream host. Replies return on the same connection.

Mechanisms

M1

Connection model

Each connection has one read goroutine and one write goroutine. Requests share a connection instead of splitting by worker. Two 8-core clients and two servers need 2 connections per server, not 16. The cost is harder coordination of shared request state.

M2

Direct registry access

Mesh-proxy connects to the service registry. This removes xDS updates that came more often than needed.

M3

Ruby balancing

Custom balancing spreads requests across Ruby replicas. This gives 15 to 20 percent more headroom before bursts overload machines.

Design decisions

No.ChoseInstead ofWhy
D1Write a new proxy in GoCustomize EnvoyThe routing change needed too much customization, and upgrade risk kept growing.
D2Use one shared connection per host pairWorker-per-core connectionsIt cuts connection counts, so services can run on smaller machines.
D3Read the registry directlyxDS updatesxDS sent extra updates that used CPU and memory.

Key numbers

  • Mesh-proxy uses about half the CPU of Envoy under high load.
  • Latency fell by roughly 50%, saving about 3,000 days of cumulative latency each day.
  • The example fell from 16 connections per server to 2.
  • Ruby services gain 15 to 20 percent more headroom.

Failures and limits

  • A 2024 Envoy upgrade pulled in a newer nghttp2 with a rapid reset defense. Default settings rejected some real traffic.
  • QA load tests caught it before production. Stripe now owns the proxy and must test it itself.

Lessons you can use

  • Count connections per core when you choose a worker model. Load test every proxy upgrade, because defaults change.
  • Measure latency and CPU against the old proxy before you commit.

Glossary

xDSThe set of Envoy APIs that push config and endpoint updates to proxies.
HedgingSending a second copy of a slow request to another host.
GoroutineA lightweight thread managed by the Go runtime.