Anurag’s Read

Selective Test Execution at Stripe: Fast CI for a 50M-line Ruby monorepo.

You see how to pick tests from runtime file access instead of static analysis, with real numbers and guardrails for the gaps. The ideas apply to any large repo in a dynamic language.
Score
9.0
Verdict
Must read
Blog and date
Stripe,
Original length and level
11 min, advanced
Topics
testing, ci, monorepo, developer productivity
Read the original

Context

The monorepo has about 50 million lines of Ruby and roughly 100,000 test files. It holds about 1.2 million test units, which would take about four months to run in sequence. Stripe runs about 50,000 builds each week, and AI-written commits will raise that number.

Running everything on each build wastes compute and slows developers.

Architecture

Tests run under a wrapper that turns on tracing through an in-house C++ library named file_access_interceptor. The library hooks open syscalls and writes each file-to-scope record to a per-process log in a temp directory. Workers stream results to a scheduler sidecar, which converts the logs and builds a global inverted index from file to scopes.

At build end, the compact index is saved as the baseline for later builds. A new build finds changed files, looks up their indexes, and unions the matching test sets. Guardrail rules then add extra tests, and linters get the list of changed files.

Mechanisms

M1

Runtime file tracing

LD_PRELOAD loads the interceptor into every test process and its children, so forked work stays attributed. The test runner sets a scope per test file through an external API. The hot path only appends a record, and all indexing happens later, outside the syscall.

M2

Hierarchical scopes

Every process starts in a root scope named r, and scopes are strings joined with //. A test selects on files opened in its own scope and in its parents. Files opened in root scope are global dependencies, so changing them can trigger many tests.

M3

Selection index

The index is a Roaring bitmap structure keyed by a global file number. Each value lists the test numbers to run when that file changes. It holds about 3 billion data points, and selection is a union of bitmaps.

M4

Change detection with hashes

Stripe hashes files with hashdeep and compares them with a stored file inventory. The inventory covers generated files that git does not track.

Design decisions

No.ChoseInstead ofWhy
D1Record file opens at runtimeStatic dependency analysisRuby dynamism and non-Ruby inputs such as YAML and fixtures make static models miss or overselect.
D2Trace at file levelPer-line execution tracingLine tracing is noisy and costly, while file opens are cheap and give enough signal.
D3Hash files for change detectionGit diffGit does not see generated artifacts.
D4Use revision IDs for orderingGit history lookups at build startLookups are slow and can fail, and IDs make selection reproducible.

Key numbers

  • About 5% of tests run per build on average, and the median is 0.5%.
  • Compute use is under 10% of a run-everything build.
  • A typical change runs a few hundred tests.

Failures and limits

  • File reads that bypass open can escape the interceptor.
  • Tests that use globbing can change behavior when a file is added, so a small set always runs.
  • Previously failing tests always rerun.
  • Root-scope changes can trigger broad runs, and a stale baseline weakens selection.
  • The post gives no false-negative rate or interceptor overhead.

Lessons you can use

  • Pick a dependency signal that survives language dynamism, such as file access.
  • Instrument at a cheap boundary and move heavy work off the hot path.
  • Make selection reproducible with an ordered baseline.
  • Add explicit guardrails for the cases where the signal is blind.

Glossary

LD_PRELOADA Linux feature that loads a chosen shared library into a program before others.
Roaring bitmapA compressed set of integers that supports fast union.