Anurag’s Read

Netflix Stratum Media Processing: Automated Container Right-Sizing.

A full case of automating a risky change: generate, verify in two stages, apply in small steps, then watch and auto-revert. The incident and its fixes teach safe rollout of resource cuts.
Score
8.6
Verdict
Must read
Blog and date
Netflix,
Original length and level
14 min, intermediate
Topics
cost, rollout, autoscaling, reliability
Read the original

Context

Stratum is Netflix's serverless platform for encoding, packaging, inspection, and quality scoring of media. Owners write CPU, memory, disk, and network sizes in YAML. These values are often too large or drift over time. Manual downsizing takes days of watching and guessing, and a bad cut causes incidents.

New formats such as live, sports, and ads make footprints hard to predict, and memory costs are rising.

Architecture

Each container reports execution data and sub-minute usage through Keystone, the event streaming platform. Summaries go to Elasticsearch for fast lookups, and raw metrics go to Iceberg for months of history. A daily Conductor workflow reads one month of Iceberg data and creates a recommendation for each function and resource type.

Each recommendation moves through five states: created, evaluating, applied, rejected, and monitored. A two-stage verification workflow tests the recommendation on replayed and live traffic. If both stages pass, the config is applied and a 90-day monitor starts.

A retry watcher inside tuned containers protects runs that exceed the history.

Mechanisms

M1

Recommendation rules

Only high-throughput jobs that tolerate delay are tuned, and they use 70 to 80% of resources. CPU and network target p99 usage, because a saturated CPU only slows a job. Memory and disk target p99.9 plus a buffer, because running out is a hard failure. Small reductions are dropped, and memory and disk cuts are capped at 15% per cycle.

Once a recommendation advances, its value and stats do not change.

M2

Retry with more resources

A watcher flags a run when non-cached memory passes 85% or disk passes 95%. The system cancels the run and requeues it on a container 1.5 times larger. This applies to memory and disk only. Thresholds are a stopgap because real out-of-memory events are hard to detect in the current design.

M3

Two-stage verification

The synthetic stage replays identical requests to a current cluster and a recommended cluster. It usually takes 1 to 2 hours. The production stage sends about 90% to the main cluster, 5% to a baseline cluster, and 5% to an experimental cluster. It compares against the baseline cluster to keep traffic comparable, and runs from 1 day to 1 week.

Checks run in order: retry rate under 1% for memory and disk, then median runtime within 10 to 15% slowdown.

M4

Post-apply monitoring

A fast 30-minute window warns at 3% retries and reverts at 10%. A slow 24-hour window alerts and reverts at 2%. Normal retry rates are about 0.1%, and up to 2 to 3% counts as ordinary variance. A revert restores the earlier baseline config with simple logic and no dynamic tuning.

Design decisions

No.ChoseInstead ofWhy
D1Verify in a synthetic stage before productionBad configs fail cheaply on replayed traffic before they touch real requests.
D2Compare against a baseline cluster, not the main clusterCompare with the main clusterA same-size baseline sees a similar traffic profile, so differences come from the config.
D3Cut memory and disk in 15% steps with a cooldownOne large cutA single 50% cut passed tests and later failed after a traffic shift.
D4Add a global kill switchOperators need one action to stop all automated right-sizing during an incident.

Key numbers

  • Hundreds of thousands of containers run each day.
  • Goal: about 10% lower total Stratum compute cost.
  • Example encoder: CPU from 8 to 4, memory from 18,000 MB to 15,300 MB.
  • Retry container size: 1.5 times the original.
  • Production experiment: 90% main, 5% baseline, 5% experimental traffic.
  • Incident: retries rose above 60% after a 50% memory cut.
  • Results: encodeV1 saved 50% CPU and 40% memory; encodeV2 settled at 20% memory.

Failures and limits

  • The encodeV2 job passed verification with a 50% memory cut and ran well for two weeks.
  • A traffic shift then pushed retries above 60%, and early versions had no auto-revert.
  • Engineers fixed it by hand, which was slow.
  • Limits remain: only high-throughput jobs are covered, retry triggers are proxies, and low traffic delays experiments.

Lessons you can use

  • Treat every automated cut as a staged change with a test, an apply step, and a monitor.
  • Limit step size, because passing tests does not prove safety under later traffic shifts.
  • Add automatic revert and a kill switch before you need them.
  • Give risky runs a fallback, such as a larger retry container.

Glossary

Right-sizingMatching resource requests to what a job really uses.
TitusNetflix's container platform that runs the jobs.
p99.9The value below which 99.9% of measured usage falls.
KeystoneNetflix's real-time event streaming platform.