Anurag’s Read

Building Vectorize, a distributed vector database, on Cloudflare's Developer Platform.

Explains how a vector database stores, indexes, versions and updates data on object storage. The snapshot and write-log design applies widely.
Score
9.0
Verdict
Must read
Blog and date
Cloudflare,
Original length and level
21 min, advanced
Topics
vector database, indexing, consistency, object storage
Read the original

Context

A vector database finds the K stored vectors closest to a query vector, with optional metadata filters. Common uses are semantic search, recommendations and retrieval-augmented generation. Normal tree indexes assume one-dimensional order, and distance math on 1,536-dimension vectors is costly.

Vectorize had to run in every Cloudflare data center, using the platform's own parts.

Architecture

A query reaches a server in the local data center, from a Worker binding or the REST API. A Rust service on that server reads index files from R2 through the Cloudflare cache. It prunes the search with the inverted file index, scores compressed vectors and refines the top results with full vectors.

Writes go to a write-ahead log that lives in a SQLite-backed Durable Object, one per index. The log stores only IDs of write payloads kept in R2, and it queues work for a shared pool of executors. An executor writes new index files and a manifest to R2, then asks the log to commit.

Commit overwrites the root manifest, and servers then read the new snapshot.

Mechanisms

M1

IVF with product quantization

Vectors are grouped by nearest centroid, and each group is stored in a file named for its centroid. A query scans only the files for centroids closest to it. Product quantization shrinks vectors for search. Full vectors are about 6 GB per million at 1,536 dimensions. The two steps give about 80% accuracy, and a refinement pass on full vectors gives over 95%.

M2

Manifests and snapshots

Each index file has its own version, and a manifest lists the versions that form one snapshot. Unchanged files carry over from the old manifest. The root manifest sits at a fixed path. An R2 PUT is atomic, so overwriting the root manifest commits the whole update at once. Old versions stay, which allows recovery to a past point.

M3

Write-ahead log and executors

A write may touch thousands of files, which is too much work for a Durable Object, so it only coordinates. It assigns exactly one executor per write and takes the commit request. If an executor stalls, the log assigns another. A late commit from the first one is rejected. Batches cover up to 200,000 vectors and 1,000 blocks. The log never waits to fill a batch.

M4

Retraining

Trainers rebuild centroids from a snapshot while writes continue. When training ends, the log applies all newer writes to the new generation. Only then does it switch queries to the new generation, so results never jump back in time.

M5

Metadata filters

Each filterable property has a sorted list of values mapped to vector ID sets. The list is split into chunks, and the manifest keeps the bounds of each chunk for binary search. The system intersects the ID sets, counts matches per centroid and scans only the best centroids.

Design decisions

No.ChoseInstead ofWhy
D1Accept eventual consistency between write and queryStrong consistencyVector indexes are rarely the main transactional store, and weaker consistency keeps queries fast and cheap.
D2Use the Durable Object as coordinator and run heavy work in executorsDo the index update inside the Durable ObjectA Durable Object has limited memory and compute for thousands of file reads and writes.
D3Index metadata per property in sorted chunksScan all metadata for each filterA scan costs too much at millions of vectors and bypasses the vector index.

Key numbers

  • Index limit grew from 200,000 vectors in beta to 5 million at general availability.
  • Vectors can have up to 1,536 dimensions, about 6 GB per million at full precision.
  • Approximate search gives about 80% accuracy and refinement gives over 95%.
  • A batch holds at most 200,000 vectors and 1,000 blocks.
  • The team loaded millions of vectors using upserts of 5,000 vectors.

Failures and limits

  • Writes become visible to queries after a delay.
  • A stalled executor can leave work half done, so the log assigns a new executor and rejects late commits.
  • The 5 million vector cap exists because building a global stateful service took time. The suggested workaround is several indexes.
  • The post does not give the centroid count, quantization settings or retraining schedule.

Lessons you can use

  • Immutable versioned files with one atomic pointer swap give cheap, safe reads at scale.
  • Keep the coordinator small and move heavy work to workers that the coordinator can replace.
  • List failure modes first, such as stalled workers, before you pick a design.
  • Batch writes by processing whatever has arrived, instead of waiting for a fixed size.

Glossary

IVFAn inverted file index groups vectors by nearest centroid so a search reads only a few groups.
Product quantizationA method that compresses vectors into short codes for fast approximate distance checks.
WALA write-ahead log records each write in order before it takes effect.
ManifestA file that lists the exact file versions of one index snapshot.