Anurag’s Read

Cutting our inference bill by 68% by being obsessive about optimization.

A model for how to investigate a cost regression with data, fix it step by step, and then prove quality did not drop. Directly useful for anyone shipping LLM features.
Score
9.0
Verdict
Must read
Blog and date
Cloudflare,
Original length and level
15 min, intermediate
Topics
cost optimization, llm, prompt caching, measurement
Read the original

Context

Cloudflare runs an AI code reviewer on GitLab merge requests. Each job starts several sub-reviewers, such as code quality, security, performance and a Codex standards reviewer. The average cost per review rose from about $1.19 in April to about $2.46 in late August. No big design change caused it. More full-tier reviews, a move to Anthropic models and more tokens per review pushed the cost up. Median review time also grew.

Architecture

A coordinator job starts for each merge request and picks a tier: trivial, lite or full. Each tier runs sub-reviewers on the OpenCode agent harness. Each sub-reviewer reads the merge request and its patches, calls tools, and posts findings. Every LLM call uses prompt caching. A cache write costs 1.25 times the base input price, and a cache read costs 0.1 times. An entry expires after five minutes without a read.

Spend split into 23% output, 43% cache writes and 34% cache reads. The code quality reviewer averaged 16.6 LLM calls per review. The team cut calls, fixed the cache prefixes, and tested models and effort levels. In October the harness moved from OpenCode v1 to v2.

Mechanisms

M1

Shared cached prefix

First requests hit the cache only 14% to 26% of the time. Yet 11k to 21k tokens of tool definitions and agent prompt were identical across repos. OpenCode v1 built one system block with the agent prompt and the environment (working directory and date). It put one cache breakpoint at the end. The team added a marker at the end of each agent prompt, and a plugin hook splits the block there. The shared half now has its own breakpoint. Repo-specific content moved after it, the skills list was disabled, and tool descriptions were frozen. First-request cache reads rose to 91% to 99.5%.

M2

Sibling prefix sharing

The code quality, security and performance reviewers send the same first request. One reviewer writes the cache entry. The other two wait up to 20 seconds for its response to start streaming, then read the entry. Each saves about nine cents.

M3

Fewer LLM calls

Sub-reviewers could see the comment-posting tool. In 57% to 76% of sessions they called it, got rejected and wasted a call. The team removed it. In 64% of code quality turns the model made one tool call. Reviewers now read shared context and patches (up to 150 KB) in the first response. About two-thirds of jobs are re-reviews, so the harness skips a reviewer when its input did not change. The harness also dropped a final summary turn that nobody read.

M4

Cancellation and timeouts

The GitLab Kubernetes executor does not signal a cancelled job. In 358 of 446 sampled cancelled jobs, the work ran to the end. Jobs now poll their status every 15 seconds. Slow reviewers used to count as provider failures and retried with the same limit. They no longer do. Security timeouts fell from 1.8% of runs to almost none.

M5

Model and effort split tests

On October 3 the team split the Sonnet reviewers 50/50 between Sonnet 5 and Sonnet 5.5. Over 2.5 days, 5.5 cost 62% less for code quality, 58% less for performance and 47% less for security. At medium effort, Sonnet 5.5 makes 1.7 calls per code quality run, against 6.2 before. High effort costs 61% more per run and gives 58% more critical and warning findings.

Design decisions

No.ChoseInstead ofWhy
D1A shared cached prefix across all reposPer-repo cachingMost prefix text is the same in every repo, so one shared entry gives a hit on the first request.
D2Sonnet 5.5 at medium effort for most reviewers, high effort for code qualitySonnet 5, and medium effort for code qualitySonnet 5.5 cost less with similar findings. Medium effort cut real code quality findings.
D3A one-step cutover to OpenCode v2 after a shadow runBoth versions agreed on the approval decision in 98% of tested merge requests.

Key numbers

  • Average cost fell 68%, from $2.43 to $0.77 per review.
  • Median review time fell from 4m09s to 1m32s. P99 fell from 15m20s to 4m41s.
  • Reviews over five minutes fell from 37.9% to 0.7%.
  • Cache hit rate rose from 85.7% to 97%. Tokens per review rose from 0.91M to 2.99M.
  • Code quality acted-on rate rose from 34% to 59%.
  • Overrated critical and warning findings fell 70% per review.

Failures and limits

  • Medium effort on performance lost about 64% of important findings. Security fell about a quarter, without a significant margin.
  • Cost per change came from weekday before-and-after comparisons, not controlled tests. Some judge groups had only 35 to 66 findings.

Lessons you can use

  • Log first-call cache usage and a hash of the system block to find why a cache misses.
  • Put shared text before repo-specific text, and give it its own breakpoint.
  • Remove tools an agent cannot use, because each wrong call costs a full round trip.
  • Test model and effort changes with a random split.

Glossary

Cache breakpointA marker that tells the provider to cache the prompt prefix that ends there.
Effort levelA model setting that controls how much reasoning the model does.
Shadow modeA run where a new system handles real input beside the old one.