Anurag’s Read

Just make it scale: An Aurora DSQL story.

A rare look at the real decisions behind a new distributed SQL database, including the rejected options and the mistakes. The Rust rewrite story gives concrete numbers for the cost and the payoff of changing language.
Score
9.0
Verdict
Must read
Blog and date
AWS,
Original length and level
15 min, advanced
Topics
databases, distributed systems, rust, postgresql
Read the original

Context

Amazon built Aurora DSQL as a relational database with no servers to manage, automatic scaling, multi-region operation, and SQL compatibility. The team solved the read path in 2021, but the write path limited scale. The first design ran on the JVM.

Simulations then showed that garbage collection pauses on single hosts ruin tail latency at scale. This pushed the team to rewrite the system in Rust.

Architecture

DSQL splits the database into components with narrow interfaces. A Postgres-based query layer parses and plans each SQL statement. An adjudicator sits in front of the journal and lets only one of several conflicting transactions win.

Each commit goes to exactly one journal, whatever number of rows it changes. The crossbar follows every journal and builds one total order by transaction time. Storage nodes subscribe to key ranges through a subscription API, and the crossbar routes committed updates to them.

Mechanisms

M1

Single-journal commit

The team rejected two-phase commit, which spreads rows over many journals and needs a coordinator. That design adds lock sequences, timeouts, liveness checks, rollbacks, and coordinator failure handling. DSQL writes each commit to one journal instead. This keeps atomicity and durability simple and lets write capacity grow when the team adds journals.

The cost moves to reads, because a reader must check every journal for the latest value of a row. Adding more journals eventually reaches network bandwidth limits.

M2

Crossbar

The crossbar separates read scaling from write scaling. Each journal is ordered by transaction time, and the crossbar merges all journals into one ordered stream. Storage nodes ask for key ranges and receive only the matching updates. Every layer fans out widely to use hardware well.

Slow subscribers force the crossbar to buffer data, and this raises garbage collection load.

M3

Tail latency simulation

The team simulated a crossbar with 40 hosts where each host sometimes stalled for one second. The model predicted about 1 million TPS. The simulation reached about 6,000 TPS. Tail latency grew from an acceptable one second to about ten seconds.

Each transaction reads from many hosts, so the chance that at least one host pauses nears 100 percent at scale.

M4

Adjudicator as the first Rust component

The team picked the adjudicator because it is simpler than the crossbar. A Rust journal client already existed, and a Kotlin version gave a fair comparison. Two engineers who were new to C, C++, and Rust wrote the Rust version. The Kotlin version, tuned over years, reached 2,000 to 3,000 TPS.

The untuned Rust version reached about 30,000 TPS. After this result the team rewrote the whole data plane in Rust.

M5

Postgres extension strategy

DSQL keeps the Postgres parser and planner. It replaces replication, concurrency control, durability, storage, and session transaction handling. The team used the public Postgres extension API so the code stays close to upstream. The first extensions used C.

A code review found several memory-safety bugs in one simple data structure, so the team moved to Rust. In Rust, one type can wrap a pointer and its length together. That type enforces the rule that links them.

Design decisions

No.ChoseInstead ofWhy
D1Rust for the data plane and extensionsJVM tuning and C or C++Garbage collection made latency unpredictable on the JVM, and C and C++ give up memory safety. Rust gives low-level control without a collector.
D2One journal per commitTwo-phase commit across journalsTwo-phase commit adds coordinator failure and liveness problems. A single journal keeps atomicity simple and moves the cost to the read side.
D3Postgres extension pointsA hard fork of PostgresA fork loses upstream features and performance work and drifts into maintenance trouble.
D4Rewrite the control plane in RustKeep Kotlin and build simulation tools for both languagesLogic copied between Kotlin and Rust diverged, and the two sides could not share test platforms.

Key numbers

  • Crossbar simulation at 40 hosts predicted about 1 million TPS and reached about 6,000 TPS.
  • Simulated tail latency grew from about 1 second to about 10 seconds.
  • The tuned Kotlin adjudicator reached 2,000 to 3,000 TPS.
  • The untuned Rust adjudicator reached about 30,000 TPS, roughly 10 times more.
  • The Rust control plane shows p99 latency close to p50.

Failures and limits

  • The Kotlin control plane was the wrong choice.
  • Shared logic lived in two languages and sometimes diverged.
  • The teams could not share test platforms, so misunderstandings caused costly debug, fix, and deploy cycles.
  • Early C extension code had memory-safety bugs.
  • It also says Rust does not fit every team, and JDK 21 suits many services.

Lessons you can use

  • Simulate tail latency before you build, because fan-out multiplies rare pauses.
  • Test a new language on one small, comparable component first.
  • Avoid splitting shared logic across two languages in one system.
  • Reuse a mature codebase through its extension API instead of a fork.
  • Give engineers time and guides when you change their language.

Glossary

JournalAn ordered, durable log where each committed transaction is written once.
AdjudicatorThe component that decides which of several conflicting transactions may commit.
Tail latencyThe slowest response times, such as the p99, which often decide user experience.
Two-phase commitA protocol where a coordinator asks all participants to prepare and then to commit a transaction.