Anurag’s Read

Behind the Scenes: Evolving Netflix's Ads Event Pipeline for Live — Part II.

A clear case study in taking a synchronous database write off the hot path and replacing it with a streaming join. The cross-region routing, dedup and shadow-traffic cutover sections transfer to almost any event pipeline.
Score
9.2
Verdict
Must read
Blog and date
Netflix,
Original length and level
14 min, advanced
Topics
streaming, system design, migrations, reliability
Read the original

Context

The Ads Event Publisher collects client ad events, adds serving context, and sends one stream to billing, metrics, and partners. Live events can bring tens of millions of viewers within minutes. The old design wrote metadata for every ad into the Ads Metadata Registry at decision time.

The registry was a globally replicated key-value store. Scaling it during an event meant rebalancing data, which could hurt the writes it served. A registry failure also stopped ad serving.

Architecture

The ad server decides an ad and logs its serving response. The device gets an opaque token that holds the serving region, and each event carries the token. The publisher is one Apache Flink job with five stages.

It first normalizes events from SVOD, live, playback, and embedded ad breaks. It then forwards each event to the region that holds its serving response. A KeyedCoProcessFunction joins the event with the response by impression key.

The job de-duplicates revenue events, extracts measurement beacons for partners, and routes enriched events downstream. An hourly Spark batch job handles events that arrive too late.

Mechanisms

M1

Keyed join with a 60-minute hold

The custom operator keeps state for each impression key. If the response arrives first, the operator enriches the event at once. If the event arrives first, the operator holds it until the response appears or 60 minutes pass. After 60 minutes it marks the event as unmatched for later recovery.

The team chose a custom operator over a built-in windowed join because ordering is handled downstream. The low-level API also allows retention tuning for each stream.

M2

Cross-region routing

A response stays in the region that served the ad. The client event can land in another region, for example after a switch from Wi-Fi to mobile data. The token carries the serving region, so the job forwards the event there. Only events that truly cross regions pay the extra hop.

M3

Late event recovery

Events arrive late when a device was offline, an ad was paused, or an engagement event was delayed. An hourly Spark job, run by Maestro, pairs these events with responses in the data warehouse. The warehouse keeps data longer than the streaming state does. The job publishes through the same sinks as the stream.

The team also uses it for dead-letter replay and backfill.

M4

In-stream de-duplication

Delivery is at-least-once, and clients retry. A stable event ID is a hash of the ad ID, the event type, and an offset or timestamp. The old design checked an external key-value store over the network for each event. The new design keys the stream by event ID, so all copies reach one task.

A local state store with a short TTL remembers recent IDs. This needs no network call and has no race. The event ID goes into the output, so billing can de-duplicate over longer windows.

M5

Rollout by parallel run

The legacy pipeline and the Flink pipeline processed the full stream and published side by side. Automated audits compared the outputs and found small differences that dashboards missed. One shared module held the enrichment logic for both systems. A stable key for each event chose exactly one publisher, and the team raised the threshold step by step.

Then the ad server stopped writing to the registry, and the team retired it.

Design decisions

No.ChoseInstead ofWhy
D1Join the event stream with the serving log in FlinkSynchronous registry writes on the serving pathThis removed the scaling risk and the critical-path dependency. The ad server already logs what it served.
D2Route each event to the region that owns its responseReplicate every region's responses to every other regionReplication costs a lot, and most responses are never used.
D3A stable-key split during cutoverA random percentage splitA random split could put the same event in both pipelines and double-count it.
D4Flink as the engineNetflix has deep platform support for it, and the team already runs the Ads Sessionizer on it.

Key numbers

  • One fight streamed to over 100 million viewers.
  • The 60-minute join window covers about 99.999% of events.
  • Parallel run agreement was above 99.99%.
  • A load test in one AWS region handled one ad break at hundreds of thousands of events each second.
  • The load test showed negligible timeouts and errors.
  • Netflix runs more than 25,000 Flink jobs.

Failures and limits

  • The post narrates no production incident.
  • Early Flink jobs had heavy, slow checkpoints and too much state in memory.
  • The team trimmed state with the platform team, but state size stays a concern.
  • De-duplication is best-effort, so duplicates can pass during reroutes, late recovery, or restart replays.
  • About 0.001% of events fall outside the window and depend on the hourly batch.

Lessons you can use

  • Remove synchronous writes from the serving path to shrink the blast radius.
  • Send events to the data instead of copying data to every region.
  • Keep the core job to one task and move other lookups off the hot path.
  • Run old and new systems side by side with shared logic and automated diffs.

Glossary

KeyedCoProcessFunctionA KeyedCoProcessFunction is a Flink operator that processes two keyed streams with shared state.
At-least-once deliveryAt-least-once delivery means a message arrives one or more times, so duplicates can occur.
TTLTTL is the time after which stored data expires.
CheckpointA checkpoint is a saved copy of job state that Flink uses to recover after a failure.