Anurag’s Read

Rebuilding Linear's delta sync read path.

A product engineer sees why index tuning and extra replicas could not fix a permission-filtered range query. The post also shows how shadow mode and a Postgres fallback made the move safe.
Score
8.8
Verdict
Must read
Blog and date
Linear,
Original length and level
7 min, advanced
Topics
sync engine, databases, performance, migrations
Read the original

Context

Linear clients keep a local database, so edits feel instant. After a reconnect, a client must fetch the changes it missed. Large workspaces make about one million sync actions per day.

The old Postgres read path made tail latency grow with workspace size.

Architecture

The app commits a sync action to Postgres. A custom CDC pipeline reads committed actions through logical replication. It writes only metadata to turbopuffer, with one namespace per workspace.

A delta request carries the ID of the last sync action the client applied. The server scans turbopuffer for ordered IDs and filters them. Surviving IDs go to Postgres in batches to fetch payloads, and enriched actions stream to the client.

Mechanisms

M1

Posting list query

turbopuffer keeps an inverted index for each filterable attribute. Each value maps to a sorted list of sync action IDs. The query unions the lists for the sync groups and subscriptions of the user. It intersects the result with the requested ID range.

M2

Two stage read

The first stage runs access checks, subscription filters, and deduplication on metadata only. The second stage fetches payloads for the surviving IDs, so Postgres no longer decodes JSON for rejected actions.

M3

Freshness and safety

Postgres serves a small authoritative slice at the head of the log, and turbopuffer serves the older range. The ranges overlap and the server deduplicates by sync action ID. If turbopuffer cannot cover a range, the request falls back to Postgres. Shadow mode compared both paths before cutover.

Design decisions

No.ChoseInstead ofWhy
D1Move intersections to posting lists in turbopufferMore index tuning and read replicasThose options leave the query shape and its CPU cost unchanged.
D2Use posting listsA Postgres GIN indexThe write cost of GIN was too high at Linear scale.

Key numbers

  • Large workspaces make about one million sync actions per day.
  • Delta sync covers more than 20 TB of sync actions.
  • Replication lag is about 1 second at p50 and a few seconds at p95.

Failures and limits

  • The old path had volatile tail latency and replicas that did not help.
  • Replication delay means recent actions must come from Postgres.

Lessons you can use

  • Move set intersection into an index built for it when filters multiply.
  • Keep large payloads out of the search store and fetch them late.

Glossary

CDCChange data capture, a pipeline that copies committed database changes to another system.
Posting listA sorted list of document IDs that match one value in an index.