Anurag’s Read

Session revocations at scale.

A full design story with a rejected option, a compact data format, a concurrency solution and measured results. The lessons on prototyping and constant factors apply to many systems.
Score
9.1
Verdict
Must read
Blog and date
Canva,
Original length and level
10 min, advanced
Topics
caching, object storage, concurrency, performance
Read the original

Context

Every backend request must identify the user, at hundreds of thousands of checks per second. Canva stores encrypted session data in browser cookies, so gateways need no datastore call for most checks. Session revocations still need a fast lookup.

Each gateway pod kept an in-memory set and filled it from MySQL at start. Hundreds of pods starting during a deploy caused a coordinated stampede on the database.

Architecture

A MySQL table is still the first record of each revocation. An asynchronous worker scans for revocations not yet uploaded. The worker writes them into chunk objects in an S3 bucket, and one chunk covers 30 minutes.

ZooKeeper elects one active worker. At start, each gateway pod downloads the chunks for the last 12 hours. Afterward the pod syncs with conditional GETs and checks each request against its in-memory set.

Mechanisms

M1

Compact record format

Each revocation has a principal and a login time cutoff. It applies to sessions that started before the cutoff. A record packs into 16 bytes. A few reserved bits mark other types, such as a brand revocation.

M2

Sorted chunk arrays

A chunk is a flat array of records sorted by principal. Gateways run a binary search directly on the downloaded bytes. A gateway lists keys after a cutoff in sorted order and fetches only recent chunks.

M3

Worker write path

The worker fetches a large batch of revocations not yet uploaded. It fetches the latest chunk, or makes a new one when enough time has passed. It inserts the new records into the sorted array and uploads the chunk again. Building a chunk of N revocations costs about O(N squared), because each batch reprocesses the whole chunk.

The worker still handled more than 2,000 revocations per second.

M4

Conditional PUT concurrency

Each update asserts that the chunk is unchanged since the worker read it. A new chunk asserts that no other process made it first. This blocks lost updates when a paused worker wakes after a new leader wrote.

M5

Gateway read path

Gateways use conditional GETs, so S3 sends a chunk only if it changed. A 30-minute chunk with 1 million revocations is about 16 MB.

Design decisions

No.ChoseInstead ofWhy
D1Store chunks in S3More MySQL read replicas, slower deploys, and a Redis cacheReplicas only delay the problem. Redis is usually not fully durable, the team would operate it, and keeping it consistent adds complexity.
D2Use a single leader workerHorizontal scaling of workersMore writers would bring back lost updates.

Key numbers

  • Records are 16 bytes each.
  • Memory use fell by 8 times, or 87.5%.
  • The worker exceeded 2,000 revocations per second.
  • One chunk of 1 million revocations is about 16 MB.

Failures and limits

  • Gateway startup downloads tens of megabytes per pod, and tens of gigabytes across the fleet.
  • The worker cannot scale out without new locking work.

Lessons you can use

  • Test real prototypes at full scale before you trust a theoretical concern.
  • Use storage conditions such as conditional PUT for correctness, and use leader election only to reduce contention.
  • Pack hot data into flat sorted arrays.

Glossary

Conditional PUTAn S3 write that succeeds only if the object still has the expected version.
ZooKeeperA coordination service that teams use for leader election.