Anurag’s Read

You've Got (Too Much) Mail: Behind the Scenes of the 3/25/26 Voice Outage.

A detailed postmortem that follows one config change through a chain of overloaded services, with a minute-level timeline and fixes. Anyone running actor-style or microservice systems will recognize the failure modes.
Score
9.0
Verdict
Must read
Blog and date
Discord,
Original length and level
14 min, advanced
Topics
incident postmortem, elixir, reliability, kubernetes
Read the original

Context

Discord's Realtime Infrastructure team was moving Elixir services to Kubernetes. The session service migration was almost done. Each connected device has a session process, and all real-time pushes to clients pass through it.

On March 25, 2026, a change to the session service dropped 17% of sessions. The reconnect load overwhelmed the voice syncers, and voice was degraded from 12:13 to 15:30 PDT.

Architecture

Gateways handle WebSocket traffic and monitor sessions. A session holds a voice state when its user is in a call. The services discord_guilds and discord_calls forward voice states to the voice syncers.

Voice syncers decide where each call runs. They send RPCs over HTTPS to a global fleet of more than 25,000 SFUs. Syncers find each other through etcd and place entities on a consistent hash ring.

The Holster library pools the HTTP connections of the gun client.

Mechanisms

M1

Session drop

The change raised pod size and cut the replica count, to test CPU overhead per pod and per session. At 12:13 PDT, Kubernetes ended 50% of the pods in the first zone. A safety check delayed the handoff of sessions, and the grace period ended first. Sessions stopped without a graceful handoff.

The service runs in three equal zones, so this was 17% of all sessions. Each session had many monitors, so the exits sent a flood of DOWN messages.

M2

Gateway reconnect storm

Gateways tell clients to reconnect when a session fails. Clients first try to resume on a gateway in the same zone, and then create a new session. The start-session rate limit was not retuned after the migration. More pods allowed more concurrent starts per host, and gateway memory in us-east1-b spiked.

The nodes restarted, and clients failed over to us-east1-c and us-east1-d, which made a second wave.

M3

Supervisor mailbox blockage

Stop and create RPCs from disconnects and reconnects caused a surge of new HTTPS connections. Each new connection passes through two single-process supervisors, Holster.Pool and the gun supervisor. A supervisor scans its mailbox for the child acknowledgement when it spawns a child. In tests, a mailbox of about 100,000 messages added about 1 ms to each spawn.

At 1 million messages and 100 spawns per second, the supervisor can never catch up. Recon showed that the Holster.Pool supervisor had the largest mailbox.

M4

etcd registration coupling

The etcd registration refresh also passed through the blocked supervisors. Instances missed the 60 second TTL and left the hash ring. The ring places each entity on a primary, a secondary, and a tertiary instance. One healthy instance carried a 3/15 share and briefly restored endpoint creation.

M5

Recovery sequence

Restarts at 12:43 and 12:47 gave brief recovery, and retries flooded the service again. A full cluster restart at 13:05 hit a thundering herd on cold start. At 13:43 the team lowered spawn rate limits in calls, streams, and guilds. A targeted restart at 14:03 under the lower limits gave full recovery.

At 14:15 the team added 15 instances, and the last backed-up instance recovered at 14:26.

Design decisions

No.ChoseInstead ofWhy
D1A validating admission webhook that blocks scale-down until handoff finishesIt prevents the original failure.
D2PartitionSupervisor for Holster.PoolA single supervisorPartitions spread connection requests over independent supervisors.
D3Move the gun connection lifecycle under Holster.PoolA separate gun supervisorThis removes the second bottleneck.
D4Lower upstream spawn rate limitsTargeted and full restarts aloneRestarts failed while the load stayed high.

Key numbers

  • The change dropped 17% of sessions.
  • Voice was degraded from 12:13 to 15:30 PDT.
  • The SFU fleet has over 25,000 hosts.
  • There were 15 syncer instances before the recovery.
  • The etcd TTL is 60 seconds.
  • A mailbox of about 100,000 messages added about 1 ms to each spawn.
  • The team added 15 instances, which halved the syncers per instance.

Failures and limits

  • A safety check and the grace period raced, so sessions stopped without handoff.
  • The rate limit was never retuned after the Kubernetes migration.
  • Full restarts caused a thundering herd.
  • The syncers scaled slowly because VMs came from Terraform, Salt, and manual etcd setup.
  • The post gives no exact new limits or user impact figures.

Lessons you can use

  • Retune rate limits when you change the deployment shape.
  • Avoid single-process choke points in the connection path.
  • When load exceeds capacity, add supply or cut demand.
  • Watch process mailbox sizes.

Glossary

MailboxThe queue of messages that an Erlang process has not yet handled.
Selective receiveAn Erlang pattern where a process scans its mailbox for a matching message.
Thundering herdA surge of clients that all retry at the same time.
Consistent hash ringA way to assign entities to instances so few move when instances change.
Grace periodThe time Kubernetes waits before it kills a stopping pod.