Anurag’s Read

Tracing Discord's Elixir Systems (Without Melting Everything).

A hands-on account of adding distributed tracing to a message-passing system where one event fans out to thousands of recipients. The cost of tracing, how the team measured it, and how they avoided it are the main lessons.
Score
8.7
Verdict
Must read
Blog and date
Discord,
Original length and level
12 min, advanced
Topics
distributed tracing, elixir, performance, observability
Read the original

Context

Discord runs each guild and each user session as an Elixir process. Metrics and logs show how often each action runs and how long it takes, but not what users felt. An internal tool called guild timings keeps per-minute action times in memory.

It rotates data fast and does not show downstream effects. Distributed tracing could close this gap, but Elixir message passing has no header layer to carry trace context.

Architecture

A client sends a message over HTTP to the Python API service. The API writes to the database and calls the guilds service over gRPC. The API already sent trace context in gRPC headers, so OpenTelemetry only needed a setting to read them.

The guild process fans the message out to session processes. Each session pushes the message to its client. Inside Elixir, a library wraps each message in an Envelope struct that carries the trace context.

Spans form a tree for each trace, and the root span decides if the trace is sampled.

Mechanisms

M1

Envelope transport library

The Envelope struct has a required message field and a trace_carrier field that defaults to an empty list. The wrap function injects the active context into the carrier with the OpenTelemetry text-map propagator. The handle function wraps plain messages in an envelope with no metadata, so old senders still work. It rebuilds the struct so fields missing in older versions get defaults.

It then extracts the context, runs the handler, and detaches the context. Drop-in call and cast functions wrap GenServer.call, with a default timeout of 5000 ms, and GenServer.cast. Old and new message formats can coexist, so services upgrade node by node.

M2

Dynamic sampling by fanout

Head sampling lets the root span decide once, and child spans inherit the decision. A message to a million users would create a million session spans. A flat second sampling layer in sessions was too aggressive, and small fanouts lost session spans. The guild now sets approximate_num_recipients on the envelope.

Sessions look up their sampling rate in a table. The rate is 100% for one recipient, 10% near 100 recipients, and 0.1% at 10,000 or more.

M3

No context for unsampled traces

Stack traces taken about every 100 ms showed busy guild processes unpacking context. Under 1% of traces were sampled, so over 99% of unpacks only learned to skip capture. The team stopped sending context for unsampled traces. An intermediate service can now start a new sampled trace, which gives incomplete traces.

M4

Root span limit in sessions

Span capture during fanout raised session CPU by 10 percentage points. The team kept capture as it was and limited where spans start. Sessions that handle post-fanout messages may continue a trace but cannot start a root span. Session CPU fell from 55% to 45%.

M5

Sampled flag check at the gRPC edge

Reading gRPC headers with OpenTelemetry doubled CPU use in the Elixir service. An experimental OpenTelemetry profiler showed 75% of gRPC handling time in context unpacking. In one profile, 14.9 of 18.9 traced seconds went to extraction. The fix reads the last byte of the encoded context, which holds the trace flags.

If the least significant bit is 0, the trace is unsampled and the code skips the full unpack.

Design decisions

No.ChoseInstead ofWhy
D1An Envelope with drop-in call and cast wrappersManual packing of context into every messageManual encoding is error-prone, and developers should not pack or unpack context by hand.
D2Sampling rate that falls as fanout size growsA flat second-layer sample in sessionsA flat rate was too aggressive for small fanouts and too costly for large ones.
D3Skip context propagation for unsampled tracesThe unpack cost on over 99% of messages was too high for guild CPU.
D4Forbid new root spans in post-fanout sessionsCheaper span captureCapture cost at fanout scale was the problem, and limiting spans cut the cost most.

Key numbers

  • One example trace showed dispatch_message at 1.69 ms and the session fanout at 357 microseconds.
  • Fewer than 1% of traces are sampled.
  • Sampling rates run from 100% for one recipient to 0.1% for 10,000 or more.
  • Session CPU fell from 55% to 45% after the root span limit.
  • gRPC context unpacking took 75% of handling time before the flag check.
  • A degraded guild took almost 16 minutes for member sessions to connect.

Failures and limits

  • Skipped propagation can give incomplete traces when a later service starts a new sampled trace.
  • Sessions after a fanout cannot start root spans, so some slow paths stay unseen.
  • Tracing cannot observe its own overhead.
  • The team reports that integration was harder than in a normal HTTP microservice.

Lessons you can use

  • Add a version-safe wrapper for context so old and new senders can run together.
  • Profile the tracing code itself, because unpacking context can cost more than the work it traces.
  • Make the sampling rate depend on fanout size.
  • Check the sampled flag first and skip all later work when it is off.

Glossary

SpanA timed region of code that links to a parent span.
Head samplingA method where the first span decides if the whole trace is kept.
GenServerAn Elixir process type that holds state and handles call, cast, and info messages.
FanoutThe act of sending one message to many recipients.
CarrierThe data structure that holds serialized trace context while it travels between services.