Anurag’s Read

Netflix's Distributed Counter Abstraction.

Shows five counter designs and why four fail. You learn how event logs and an immutable time window give idempotent counts without locks.
Score
8.8
Verdict
Must read
Blog and date
Netflix,
Original length and level
19 min, advanced
Topics
counters, cassandra, idempotency, distributed systems
Read the original

Context

Netflix teams need to count user actions, feature exposures and experiment metrics at large scale. Some uses accept approximate and non-durable counts. Others need accurate and durable counts with a small delay. Both kinds need high throughput, low latency and high availability across regions.

A plain counter row in a replicated store causes write contention and gives no safe retries.

Architecture

Clients call an API like an atomic integer: add, add and get, get, and clear. Each write carries an idempotency token. The best-effort counter uses EVCache keys with increment, decrement and TTL operations in one region. The eventual counter writes each change as an event into the TimeSeries store, backed by Cassandra.

After a durable write, the service sends a light rollup trigger that holds only the namespace and counter name. Triggers go into in-memory queues on each instance. A hash of the counter name picks the queue. A consumer drains a queue into a set, scans new events since the last checkpoint, and sums them.

The sum and checkpoint go to a Rollup table in Cassandra and to an EVCache entry. Reads return the last rolled-up value.

Mechanisms

M1

Event store and idempotency

Each increment is stored with event time, event ID and item key. Together they form the idempotency key. Time buckets and event buckets keep Cassandra partitions narrow. Events sort in descending time order, which helps with resets. A retention policy deletes events after they are aggregated and archived.

M2

Immutable window rollup

Each counter has a last rollup timestamp. Rollups only cover events older than now minus a safe lag. The TimeSeries service rejects events beyond an accept limit, so the window stays closed. The lag also covers clock skew and is usually a few seconds. Overlapping rollups write the same sums, so no distributed lock is needed. Values may flicker briefly, then converge.

M3

Queue routing and rebalancing

XXHash maps each counter to a fixed queue, so duplicate work stays low. A separate stack runs fewer and larger instances for rollups. Operators change the queue count, for example 8 to 16, in control plane config and redeploy. Old queues drain on deploy, and brief overlap between old and new servers is accepted.

M4

Adaptive batching and back-pressure

Consumers process a batch in parallel with range scans on the event store. The number of time partitions per scan changes with counter cardinality. The consumer waits for a batch to finish and adjusts the pause from the last batch's speed. Low-cardinality counters stay in the rollup loop. High-cardinality counters re-queue only when the last write time shows new events.

M5

Cache of count and checkpoint

EVCache stores the rolled-up count and its timestamp as one value, so they cannot disagree. A read also triggers a rollup. This speeds later reads and repairs earlier failed rollups.

Design decisions

No.ChoseInstead ofWhy
D1Log every increment as an event and roll up in the backgroundOne row per counter, bucketed keys, in-memory aggregation, or Kafka partitions onlyEvents allow audit and recount and give idempotency. The other options had contention, data loss on crash, or no way to recount.
D2Use EVCache for the best-effort counterHyperLogLog and Count-Min SketchHyperLogLog counts distinct items, not signed increments. Count-Min Sketch cannot reset or expire keys easily. Netflix also wanted to reuse stores it already runs.
D3In-memory queues for rollup triggersDurable queuesThey are cheap and easy to rebalance. Lost triggers are repaired by later reads.

Key numbers

  • About 75K count requests per second globally.
  • Single-digit millisecond latency on all endpoints.
  • Example config: 8 queues, 10 second coalescing, 16 MiB per queue, rollup batch of 32.
  • Event buckets use 600 second buckets and 4 buckets per ID.
  • Accept limit of 5 seconds; events deleted after 7 days.

Failures and limits

  • Best-effort counters have no idempotency, so retries can double count. They also do not replicate increments across regions.
  • If an instance crashes, queued rollup triggers are lost. Rarely read counters may stay stale until the next read.
  • Cross-region replication problems can cause missed events, and clearing a counter across regions is still hard.
  • The accurate counter adds the unrolled delta at read time. It costs more as the number of events grows, so it stays experimental.

Lessons you can use

  • Store events instead of totals when you need idempotency and the option to recount.
  • Aggregate only data that can no longer change. A closed time window removes the need for locks.
  • Make aggregation safe to repeat, so overlapping workers and retries cause no harm.
  • Let reads repair the system by triggering work. This heals missed background jobs cheaply.

Glossary

EVCacheNetflix's distributed cache built on Memcached.
TimeSeries AbstractionA Netflix service that stores timed events on top of Cassandra.
Idempotency tokenA unique ID that lets the server ignore a repeated request.
RollupA background sum of raw events into one stored count.