Anurag’s Read

How We Moved Discord Voice to the Edge.

A candid migration story with failed rollouts, reverted regions, and low-level debugging from packet loss to scheduler starvation. It teaches how to pace a risky infrastructure move with metrics and a hosting partner.
Score
8.9
Verdict
Must read
Blog and date
Discord,
Original length and level
13 min, advanced
Topics
migrations, networking, real-time media, performance
Read the original

Context

Voice latency depends on the distance between users and the voice server. Discord ran voice in about 30 cities, where large cloud vendors had data centers. The team wanted Cloudflare, which runs in more than 300 cities.

Each call runs on a single host, so a wrong host placement can hurt users.

Architecture

A Selective Forwarding Unit (SFU) hosts one whole call on one instance. Every participant sends packets to that host and receives packets from it. Each voice host is a container on Cloudflare with worker threads.

The host has a stateful control plane for membership, mute state, and stream routing. It also has a stateless media plane for UDP audio and video packets. On startup a host registers itself in a discovery service and refreshes the record on a timer.

The records live in Valkey on GCP Memorystore with a 10-minute TTL, so dead hosts expire.

Mechanisms

M1

Host self-registration

Cloudflare creates and reclaims hosts on its own schedule, so provisioning cannot register them. Each host registers on startup, re-registers on a timer, and unregisters on shutdown. The older etcd store struggled at about 25,000 Cloudflare voice hosts. The team wrote records to both stores during the cutover and then retired etcd.

In a coordinated recreate, the supervisor waits five minutes after the shutdown signal. This delay lets replacements start, so a point of presence never drops to zero instances.

M2

Worker density tuning

Each Rust SFU worker has its own UDP socket and a single-threaded Tokio runtime with a select loop. In May 2025, US East showed 1.5% to 2% packet loss, against under 0.5% on the old vendor. All workers shared one NIC transmit queue, and the kernel dropped UDP packets at the send buffer. The team cut each host to two workers, and loss matched the old vendor.

After a Cloudflare buffer fix, eight workers became the global default in late September 2025.

M3

Write starvation fix

During the European ping spikes, the receive future stayed ready while the receive buffer held data. The select loop kept choosing receive, so the flush timer never ran. The Tokio budget of 128 polls ran out in microseconds at peak, so it gave no protection. Send queue latency rose to hundreds of milliseconds, against a baseline near 1 ms.

The fix disabled receive for one poll when 9 ms had passed since the last flush. This forced the flush timer to run, and p99 improved.

M4

Softirq pinning

Each VM had one virtio-net queue, so all receive softirq work ran on one vCPU. Worker threads on that vCPU were preempted in the middle of their work. The team used taskset in the container entrypoint to keep workers off the IRQ vCPU. Receive Packet Steering moved softirq work to other cores.

The change shipped on March 11, and the peak spikes ended.

Design decisions

No.ChoseInstead ofWhy
D1Replace existing regions one at a timeAdd hosts in regions with no coverage, such as ReykjavikA local host helped Iceland-only calls but hurt mixed-region calls, because one host serves the whole call.
D2Hosts register themselves in Valkey with a TTLRegistration at provisioning time, and etcdCloudflare creates and reclaims hosts unpredictably, and etcd strained at about 25,000 hosts.
D3Check peering headroom with major ISPs before moving trafficRotterdam traffic moved to Amsterdam and crossed a saturated Telia handoff for Orange users.

Key numbers

  • More than 80% of voice and video traffic runs on Cloudflare.
  • Iceland-only calls had ping down 9% and packet loss down 11%.
  • Frankfurt ping fell 34% and packet loss fell 42% against the old vendor.
  • Orange users saw call latency above one second and a 30% worse freeze ratio, so the team reverted.
  • Packet loss fell 20% to 60% in several European and Latin American regions.

Failures and limits

  • The Rotterdam move failed for Orange users, because the path crossed a saturated transit link at peak.
  • A page cache flush of several gigabytes stalled other writers on the block device.
  • Cloudflare fixed that with more frequent dirty page flushing and a VMM sync I/O change.
  • Reykjavik and Auckland remain open, and the affinity workaround stays until Cloudflare moves to multi-queue virtio-net.

Lessons you can use

  • Ramp traffic slowly and watch quality metrics at each step.
  • Check network peering for the main ISPs of a region before you shift users.
  • A select loop that always picks a ready branch can starve a timer, so add an explicit time budget.
  • Share telemetry with the vendor, because some faults sit below your own code.

Glossary

SFUA server that receives media from each call participant and forwards it to the others without mixing it.
SoftirqA Linux kernel mechanism that handles deferred work, such as network packet receive, on a CPU.
Receive Packet SteeringA Linux setting that spreads packet processing across several CPU cores.