M1Session drop
The change raised pod size and cut the replica count, to test CPU overhead per pod and per session. At 12:13 PDT, Kubernetes ended 50% of the pods in the first zone. A safety check delayed the handoff of sessions, and the grace period ended first. Sessions stopped without a graceful handoff.
The service runs in three equal zones, so this was 17% of all sessions. Each session had many monitors, so the exits sent a flood of DOWN messages.
M2Gateway reconnect storm
Gateways tell clients to reconnect when a session fails. Clients first try to resume on a gateway in the same zone, and then create a new session. The start-session rate limit was not retuned after the migration. More pods allowed more concurrent starts per host, and gateway memory in us-east1-b spiked.
The nodes restarted, and clients failed over to us-east1-c and us-east1-d, which made a second wave.
M3Supervisor mailbox blockage
Stop and create RPCs from disconnects and reconnects caused a surge of new HTTPS connections. Each new connection passes through two single-process supervisors, Holster.Pool and the gun supervisor. A supervisor scans its mailbox for the child acknowledgement when it spawns a child. In tests, a mailbox of about 100,000 messages added about 1 ms to each spawn.
At 1 million messages and 100 spawns per second, the supervisor can never catch up. Recon showed that the Holster.Pool supervisor had the largest mailbox.
M4etcd registration coupling
The etcd registration refresh also passed through the blocked supervisors. Instances missed the 60 second TTL and left the hash ring. The ring places each entity on a primary, a secondary, and a tertiary instance. One healthy instance carried a 3/15 share and briefly restored endpoint creation.
M5Recovery sequence
Restarts at 12:43 and 12:47 gave brief recovery, and retries flooded the service again. A full cluster restart at 13:05 hit a thundering herd on cold start. At 13:43 the team lowered spawn rate limits in calls, streams, and guilds. A targeted restart at 14:03 under the lower limits gave full recovery.
At 14:15 the team added 15 instances, and the last backed-up instance recovered at 14:26.