Anurag’s Read

Cloudflare outage on November 18, 2025.

A clear postmortem where a harmless database permissions change doubled a config file and crashed a Rust proxy. It teaches how to treat internal config as untrusted input and how mixed good and bad states mislead debugging.
Score
9.0
Verdict
Must read
Blog and date
Cloudflare,
Original length and level
12 min, intermediate
Topics
incident, postmortem, config-safety, rust
Read the original

Context

Bot Management scores each request with a machine learning model. The model reads a feature file that a ClickHouse query rebuilds every five minutes and pushes to the whole network. The proxy preallocates memory for at most 200 features, and production used about 60.

A permissions change on the database cluster ran at the same time as the migration from the old proxy FL to the new proxy FL2.

Architecture

A request passes TLS termination, then the core proxy, then Pingora for cache or origin fetches. The core proxy runs per-customer modules such as WAF, DDoS protection and Bot Management. Distributed ClickHouse tables in the default database read from tables in the r0 database.

The feature query listed columns by table name only and ignored the database name. After access to r0 metadata became visible at 11:05, the query returned each column twice. The oversized file reached every proxy, and the Rust code failed when it exceeded the limit.

Mechanisms

M1

Gradual rollout of the permission change

The change reached ClickHouse nodes in stages. Each five-minute cycle could run on an updated node or an old one. The file was good or bad depending on that node. Once all nodes were updated, every file was bad and the failure became stable.

M2

Limit and panic

The module caps features at 200 to keep memory use fixed. FL2 called unwrap on an error result when the file was too large. This panic ended the worker and returned 5xx errors. FL customers saw no errors, but every request received a bot score of zero, which caused false positives for blocking rules.

M3

Knock-on failures

Workers KV, Turnstile and Access depend on the core proxy. Turnstile sat on the dashboard login page, so most users could not log in. A login backlog and retries caused a second dashboard dip until control plane concurrency was raised. Debug and error reporting systems used large CPU time, which raised latency.

Design decisions

No.ChoseInstead ofWhy
D1Bypass the failing proxy path for Workers KV and Access at 13:05This cut impact before the real cause was fixed.
D2Roll back to a last-known-good Bot Management file and stop generating new onesKeep debugging the live generatorStopping propagation was the fastest way to end the errors at 14:24.

Key numbers

  • Impact started 11:20 UTC, main fix 14:30, full recovery 17:06.
  • Feature cap 200, normal use about 60.
  • Feature file regenerated every 5 minutes.
  • Dashboard degraded 11:30 to 13:10 and 14:40 to 15:30.
  • Config change deployed at 11:05.

Failures and limits

  • The team first suspected a large DDoS attack.
  • The external status page went down at the same time by coincidence, which supported that theory.
  • The flip between good and bad files every five minutes made the pattern look unusual.
  • The panic was an unhandled error path, not a design for failure.

Lessons you can use

  • Treat files that your own systems generate like user input and validate size and shape.
  • Prefer a safe fallback over a panic when a limit is exceeded.
  • Keep global kill switches for features so you can disable one quickly.
  • Watch for dependency changes in a data query when you change access rules.

Glossary

ClickHouseA column-oriented database used for analytics queries.
FL2The new Rust version of Cloudflare's core proxy.
Bot scoreA number that tells how likely a request comes from a bot.
PingoraCloudflare's Rust proxy for cache and origin traffic.