Compact record format
Each revocation has a principal and a login time cutoff. It applies to sessions that started before the cutoff. A record packs into 16 bytes. A few reserved bits mark other types, such as a brand revocation.
Every backend request must identify the user, at hundreds of thousands of checks per second. Canva stores encrypted session data in browser cookies, so gateways need no datastore call for most checks. Session revocations still need a fast lookup.
Each gateway pod kept an in-memory set and filled it from MySQL at start. Hundreds of pods starting during a deploy caused a coordinated stampede on the database.
A MySQL table is still the first record of each revocation. An asynchronous worker scans for revocations not yet uploaded. The worker writes them into chunk objects in an S3 bucket, and one chunk covers 30 minutes.
ZooKeeper elects one active worker. At start, each gateway pod downloads the chunks for the last 12 hours. Afterward the pod syncs with conditional GETs and checks each request against its in-memory set.
Each revocation has a principal and a login time cutoff. It applies to sessions that started before the cutoff. A record packs into 16 bytes. A few reserved bits mark other types, such as a brand revocation.
A chunk is a flat array of records sorted by principal. Gateways run a binary search directly on the downloaded bytes. A gateway lists keys after a cutoff in sorted order and fetches only recent chunks.
The worker fetches a large batch of revocations not yet uploaded. It fetches the latest chunk, or makes a new one when enough time has passed. It inserts the new records into the sorted array and uploads the chunk again. Building a chunk of N revocations costs about O(N squared), because each batch reprocesses the whole chunk.
The worker still handled more than 2,000 revocations per second.
Each update asserts that the chunk is unchanged since the worker read it. A new chunk asserts that no other process made it first. This blocks lost updates when a paused worker wakes after a new leader wrote.
Gateways use conditional GETs, so S3 sends a chunk only if it changed. A 30-minute chunk with 1 million revocations is about 16 MB.
| No. | Chose | Instead of | Why |
|---|---|---|---|
| D1 | Store chunks in S3 | More MySQL read replicas, slower deploys, and a Redis cache | Replicas only delay the problem. Redis is usually not fully durable, the team would operate it, and keeping it consistent adds complexity. |
| D2 | Use a single leader worker | Horizontal scaling of workers | More writers would bring back lost updates. |
| Conditional PUT | An S3 write that succeeds only if the object still has the expected version. |
|---|---|
| ZooKeeper | A coordination service that teams use for leader election. |