Anurag’s Read

Strobelight: A profiling service built on open source technology.

A rare look at fleet-wide profiling design: sampling, symbolization, safeguards and a one-character fix worth thousands of servers. Strong lessons for anyone who runs production services.
Score
8.6
Verdict
Must read
Blog and date
Meta,
Original length and level
14 min, advanced
Topics
profiling, performance, ebpf, infrastructure
Read the original

Context

Meta runs very large fleets, and small inefficiencies cost many servers. Static analysis cannot see fleet-wide compute cost. Slow regressions can go unseen until traffic is huge.

Architecture

Strobelight schedules profilers on hosts from configuration in Configerator. Default profilers run on every host at tuned intervals. Engineers can also start profiles from a CLI or web UI.

Raw samples go to disk, then a central symbolization service adds names. Data lands in Scuba, and tools such as Tracery show it as flame graphs and timelines.

Mechanisms

M1

Dynamic sampling

Each service has a target number of samples per hour. Strobelight starts with a conservative run probability and picks times at random. Fixed times such as noon would hide traffic patterns. The next day it checks actual yield and adjusts the probability.

Each sample carries a weight so results compare across services.

M2

Safeguards

Rules stop profilers from overloading hosts or storage. Mutual exclusion prevents two profilers from using the same hardware counter. A queue limits concurrency. Owners can still collect heavy data when they debug.

M3

Central symbolization

Debug data can reach gigabytes, so hosts do not parse it. A service pre-processes binaries and stores the needed data. User-space work runs after profiling to avoid dropping eBPF samples. All binaries need frame pointers.

M4

Stack schemas and Strobemeta

A small language tags stacks and removes unwanted frames. Teams build dashboards of expensive copies or container use. Thread-local metadata links stacks to endpoints or request IDs.

Design decisions

No.ChoseInstead ofWhy
D1Randomized, tuned sampling per serviceFixed schedules and static ratesFixed times add bias, and static rates give uneven sample counts.
D2Central symbolizationSymbolizing on each hostLarge debug data risks memory pressure on production hosts.

Key numbers

  • 42 profilers exist.
  • Up to 20% fewer CPU cycles for some services.
  • All top 200 services have FDO profiles.
  • One fix saved about 15,000 servers of yearly capacity.
  • Example service: 1,000 hosts, 40,000 samples per hour.

Failures and limits

  • Samples drop when user-space processing falls behind.
  • Profilers can harm workloads without safeguards.
  • Frame pointers are required.
  • Adding a built-in profiler takes weeks, so bpftrace scripts fill the gap.

Lessons you can use

  • Keep always-on profiling so incidents are never diagnosed blind.
  • Randomize sampling and record weights to avoid bias.
  • Move heavy processing off production hosts.
  • Fleet-wide data finds waste that code review cannot.

Glossary

eBPFA Linux feature that runs small programs in the kernel with low overhead.
SymbolizationTurning instruction addresses into function names.
FDOFeedback-directed optimization, which uses runtime profiles to improve compiled code.