Anurag’s Read

How Discord Indexes Trillions of Messages.

A rebuild of a search system that names each failure of the old design and the matching fix, with latency numbers. The ideas on blast radius, durable queues, and zero-downtime reindexing transfer to many data systems.
Score
9.0
Verdict
Must read
Blog and date
Discord,
Original length and level
10 min, advanced
Topics
search, elasticsearch, sharding, migrations
Read the original

Context

Discord lets users search trillions of messages. In 2017 the search used Elasticsearch on two clusters, with each guild or DM stored on one index and node. Messages were indexed lazily, because not all users search.

A Redis queue fed workers that sent bulk index requests.

Architecture

Messages enter a Google Pub/Sub queue. A router reads the stream and finds the cluster and index for each message. The router sends each message to a channel for that destination.

One task per destination groups messages into a single bulk index call. Each cluster has ingest, master-eligible, and data nodes on separate node pools. Guild messages shard by guild_id, and DM messages shard by user_id in a separate cell.

Mechanisms

M1

Destination batching router

The destination key is the cluster plus the index that the message maps to. On a new key, the router creates an unbounded channel and spawns a Tokio task. A map from key to sender sits behind a read-write lock, and each send takes the write lock. Each task collects chunks and issues one bulk call.

A node failure now hurts only the messages for that node. Before, a failed node failed about 40% of batches, by the post's estimate.

M2

Cells and node roles

Master nodes ran out of memory as node and index counts grew, which caused indexing failures and query timeouts. The team keeps each index near Elastic guidance of 200 million messages and 50 GB. Each cluster has 3 master-eligible nodes and at least 3 ingest nodes, one per zone. Shard allocation awareness places a primary and its replica in different zones.

M3

Big guild reindex

A Lucene index holds about 2 billion documents at most, and large guilds approach that limit. The team creates a new index and dual-writes new messages to the old and new index. A backfill copies history into the new index while queries still use the old one. After the backfill, query traffic switches to the new index.

The team then stops writes to the old index and cleans up its data.

Design decisions

No.ChoseInstead ofWhy
D1Google Pub/Sub as the message queueRedisPub/Sub guarantees delivery and holds large backlogs, so failures slow indexing and no longer drop messages.
D2Many small clusters in cellsClusters of over 200 nodesCoordination cost grows with node and index count, and big clusters could not drain nodes for upgrades.
D3Store DMs twice, by userFan-out queries over all DM shards of a userCross-DM search needs one shard per user to stay cheap.

Key numbers

  • The new system runs 40 Elasticsearch clusters with thousands of indices.
  • Median query latency fell from 500 ms to under 100 ms.
  • The p99 query latency fell from 1 s to under 500 ms.
  • Indexing throughput doubled.

Failures and limits

  • Redis hit maximum CPU during Elasticsearch node failures and dropped messages.
  • Log4Shell needed a setting change on every node, which caused a full search outage for a maintenance window.
  • The old cluster could not take rolling upgrades, so it stayed on old OS and Elasticsearch versions.
  • The post gives no shard counts, heap sizes, or backpressure details.

Lessons you can use

  • Make each bulk write touch one failure domain, so one failure stays small.
  • Use a queue that keeps messages during backlog instead of one that drops them.
  • Set a target size per index and per cluster before the system grows.

Glossary

CellA group of several small clusters that Discord treats as one unit for a class of data.
MAX_DOCThe Lucene limit of about 2 billion documents in one index.
BFGA Big Freaking Guild, a guild with so many messages that it needs special index handling.