Anurag’s Read

Translations at Scale: Memory Optimization Techniques That Kept Uber's P99 Under 1ms.

A step-by-step Go memory case study with clear alternatives, hash-collision math, and before/after numbers. The lessons about garbage collection pressure and hybrid memory and disk storage apply to any read-heavy cache.
Score
8.8
Verdict
Must read
Blog and date
Uber,
Original length and level
8 min, advanced
Topics
golang, memory optimization, caching, performance
Read the original

Context

Uber serves millions of translations per second from memory. Garbage collection is a top compute cost in Uber Go services. In one service, over 20% of CPU went to garbage collection, and about 70% of objects came from translations.

Refreshes also doubled memory for up to a minute and caused out-of-memory kills.

Architecture

At start and at each refresh, the service downloads a compressed file and stream-parses the JSON. It calls Put for each pair on a store with 256 shards. Each shard has its own RWMutex, in-memory entries, and disk file.

A key becomes a 6-byte hash, where the first byte picks the shard. Get returns the value from memory or reads it from disk. A global time-based LRU decides which values stay in memory, and the finished store swaps in atomically.

Mechanisms

M1

Plain map

Before Go 1.24, sync.Map needed about four heap objects for each string pair. The workload had many reads, few updates, and no deletes. A plain map[string]string with an RWMutex adds no extra objects per insert. This cut service CPU by over 30%.

M2

Hashed keys

Keys used 25 to 30% of memory, so the team stored 6-byte hashes. For 10 million keys, 6 bytes gives about 0.177 expected collisions. A fallback compares the original key string on a collision. Key memory fell to about 5%.

M3

Compact entries

Each entry stores a 2-byte memory index and a 3-byte disk offset. With the key, an entry costs 11 bytes. Offsets pad to multiples of 8, so each shard addresses 128 MB.

M4

LRU clock

Each entry records its last access time. The global LRU updates only when that time is over about one second old. A system clock call cost 3 to 5% CPU, so a background goroutine keeps an atomic timestamp. A refresh preloads only entries used in the last 15 minutes.

Design decisions

No.ChoseInstead ofWhy
D1A local store with a disk partAn external serviceA service adds network overhead, latency, and failure modes.
D2One shared datasetPer-service subsetsSubsets need maintenance across many repositories.
D3A global LRUA per-shard LRUA per-shard LRU can evict from hot shards.

Key numbers

  • Resident memory fell from about 2.7 GiB to about 750 MiB.
  • Memory for the example data fell from about 800 MB to about 40 MB of metadata.
  • At most 50,000 items sit in memory, and total memory is 80 to 120 MB.
  • The hit rate is above 99.9%.
  • Disk reads take about 0.03 ms at p50 and 0.05 ms at p99.

Failures and limits

  • Six-byte hashes still collide rarely.
  • The 32 GB offset limit can break with growth or uneven shards.
  • Results cover one service and one time window, and the analysis applies to Go before 1.24.

Lessons you can use

  • Count objects as well as bytes.
  • Check how much cached data is read.
  • Keep the API fixed so storage changes stay local.
  • Measure the cost of cheap calls such as clock reads.

Glossary

LRULeast recently used, an eviction rule that removes the entry unused for the longest time.
RSSResident set size, the physical memory a process uses.
RWMutexA lock that allows many readers or one writer.