Anurag’s Read

Canva incident report: API Gateway outage.

A candid postmortem of a cascading failure that links a CDN stall, a thundering herd and a hidden lock bug. The action items show how to fix canary gaps and prepare for load spikes.
Score
9.2
Verdict
Must read
Blog and date
Canva,
Original length and level
9 min, intermediate
Topics
incident report, reliability, cdn, load shedding
Read the original

Context

On 12 November 2024, canva.com was unavailable from about 9:08 to 10:00 UTC. Three causes combined. These were an editor deployment, a telemetry lock bug in the API Gateway, and network trouble at the Cloudflare CDN. The editor is a single page app that deploys several times a day.

Each deploy publishes more than 100 static assets to an S3 bucket. The API Gateway is the entry point for all API traffic, and it handles authentication, authorization, and rate limits.

Architecture

Clients fetch editor assets through Cloudflare tiered caching, which checks the local edge, then upper tiers, then the S3 origin. After the assets load, the editor script calls the API Gateway. The gateway uses Netty and runs as an autoscaled group of tasks on Amazon ECS.

Load balancers spread requests across the tasks. Netty event loop threads must never block.

Mechanisms

M1

Stalled asset and request coalescing

A deploy at 8:47 UTC sent clients to fetch new assets. The path between the Cloudflare Singapore and Ashburn sites degraded, and one asset took about 20 minutes to download. Cloudflare merges identical requests into one origin fetch and streams the body as it arrives. More than 270,000 user requests waited on that one stream.

The canary saw no errors, because the requests had not yet finished.

M2

Thundering herd

The asset finished at 9:07 UTC, and all waiting requests completed at once. The API Gateway then received about 1.5 million requests per second, three times the usual peak.

M3

Telemetry lock contention

A recent telemetry library change re-registered some metrics on every recorded value. The re-registration took a lock inside a third-party library. Under load, the lock blocked event loop threads and cut the throughput of each task sharply. A fix was already in the release process, but the team did not rush it.

M4

Memory collapse

Backlogged requests grew, and load balancers opened new connections to tasks that were already overloaded. Off-heap memory grew until the Linux out-of-memory killer ended all containers within 2 minutes.

M5

Recovery by blocking traffic

At 9:29 UTC the team added a Cloudflare firewall rule that blocked all traffic. New tasks then started with no load. They sent canva.com to a status page. They restored traffic in stages, starting with Australian users under strict rate limits.

Design decisions

No.ChoseInstead ofWhy
D1Block all traffic at the CDN to recoverScaling the task countNew tasks died under load, so only a load free start could work.
D2Add page load completion events as a second canary signalUsing only the JavaScript error rateThe error rate stayed flat while requests were stuck.

Key numbers

  • The outage ran from about 9:08 to 10:00 UTC.
  • The p90 time to first byte rose by more than 1700%.
  • More than 270,000 requests waited on one stream.
  • Peak load was about 1.5 million requests per second, three times the usual peak.

Failures and limits

  • Cloudflare said a stale traffic rule sent IPv6 traffic over public transit, which blocked automatic rerouting. The rule is gone.
  • The canary could not see stuck requests.
  • The team underestimated the telemetry fix.

Lessons you can use

  • Watch for success signals, because a deploy can stall without errors.
  • Test shared libraries with many threads, since a lock in an event loop is costly.
  • Plan how to block load before an outage starts.

Glossary

Request coalescingA CDN feature that sends one origin request for many identical client requests.
Thundering herdA sudden burst of requests that arrive at the same moment.