Anurag’s Read

Build better software to build software better.

A strong case study on why adopting Bazel does nothing until the build graph is clean, with a clear diagnosis of cache misses and coupling. It gives numbers and rules that transfer to any slow build.
Score
9.0
Verdict
Must read
Blog and date
Slack,
Original length and level
18 min, advanced
Topics
build systems, bazel, developer productivity, caching
Read the original

Context

The backend build for Quip and Canvas took 60 minutes. Feedback came too late, so builds on each pull request were impractical and breakages reached main. The build graph had cycles, oversized steps, steps that were not idempotent, and steps that changed the working directory.

The cache hit rate was close to zero. The team adopted Bazel only after they restructured the build.

Architecture

Bazel models each build step as a target with inputs, outputs, and a command. Targets run in a sandbox that exposes only the declared inputs. Bazel caches each action and runs only the actions that the requested outputs need.

It spreads work over local cores or a build cluster. The team rewrote build orchestration from Python into Starlark and kept build logic in BUILD.bazel files. Remaining Python scripts use only the standard library.

The frontend builder now compiles one bundle from one set of sources. Bazel chooses the scope and the parallelism.

Mechanisms

M1

Cache granularity

A cache only helps when the cached unit is small enough. A function that takes a whole list misses the cache when one item changes. The same function that takes one item caches per item. Caching also needs hermetic and idempotent steps.

The old build resembled a cached function with about 100 parameters where 2 or 3 always changed. Its hit rate was close to zero.

M2

Backend and frontend decoupling

The transitive inputs of the frontend included the entire built Python backend. Every Python change altered the frontend cache key and forced a full frontend rebuild. This edge cost an average of 35 minutes per build. The team removed backend and build dependencies from the Python scripts.

Python changes no longer affect TypeScript cache keys. A full build took about 25 minutes when the frontend came from cache.

M3

Frontend builder redesign

The old builder took all TypeScript, Less, and CSS sources and settings. It parallelized over local worker processes and emitted all bundles. One changed input invalidated everything, and the builder competed with Bazel for resources. The new builder has no parallelism and builds one bundle.

TypeScript to JavaScript and Less to CSS run as separate tracks, and each output caches on its direct inputs.

M4

Rust comparison tool

Build code had almost no tests. The team wrote a Rust tool that compared artifacts from the old and new processes. Each difference pointed to a logic bug in the new build. The old output served as the definition of correct.

Design decisions

No.ChoseInstead ofWhy
D1Restructure the build before adopting BazelDrop Bazel onto the existing buildThe cache would never hit, and Bazel parallelism would duplicate the existing ad hoc parallelism.
D2Move orchestration to StarlarkPython multiprocessing and async orchestrationThat mix coupled backend, frontend, and build code.
D3Remove parallelism from the frontend builderKeep its worker-process poolThe pool limited work to one machine and competed with Bazel scheduling.
D4Cache each bundle on its own inputsOne cache unit for the whole frontendOne input change forced a full rebuild, and a single-bundle build was impossible.

Key numbers

  • The old build took 60 minutes in every case.
  • The new build takes 10 minutes at best, about 12 on average, and 30 in the worst case.
  • The headline claim is up to six times faster.
  • The Python to TypeScript edge cost an average of 35 minutes per build.
  • A full build took about 25 minutes when the frontend came from cache.
  • The old cache hit rate was close to zero.

Failures and limits

  • Cache-miss builds still take about 30 minutes.
  • Build code had almost no tests, so the old output defined correctness.
  • Before the work, engineers could not reason about blast radius.
  • The post gives no per-stage breakdown beyond the 35-minute and 25-minute figures.

Lessons you can use

  • Design build code with the same separation, granularity, and composability rules as application code.
  • Check the dependency graph for hidden edges, because one edge can cost more than half the build time.
  • Keep orchestration, logic, and parallelism in separate layers.
  • Build a comparison tool before you rewrite code that has no tests.

Glossary

HermeticDepending only on declared inputs and nothing else on the machine.
IdempotentGiving the same output every time for the same inputs.
StarlarkThe restricted Python-like language that Bazel uses for build files.
Transitive inputAn input that a target needs through another target it depends on.