Invalid states
Safety invariants mark states invalid. An even voting count is invalid. A minority of healthy voters is invalid, because it risks write availability. A node with priority 1 and zero votes is invalid.
Stripe's Document Database fleet has about 2,000 MongoDB shards and 40+ shard layouts. These shards processed $1.9T in payments in 2025. A person needed about 15 minutes to order the fixes for a degraded shard. Running the operations took up to 3 hours. The first automation was a hard-coded state machine with plugins in a fixed order. Combined failures broke it, and it paged operators.
A planner models a shard as a graph. Each node is a shard state. Each edge is one atomic operation. Operations include vote and priority changes, hide and unhide, node rebuild, and oplog resize. The start is the observed ground reality, and the goal is the Source of Truth (SOT). The planner generates successor states, drops invalid ones, and searches for the cheapest path.
Temporal runs the chosen operations as durable workflows. A CommonContext interface lets the same code run in production or in simulation.
Safety invariants mark states invalid. An even voting count is invalid. A minority of healthy voters is invalid, because it risks write availability. A node with priority 1 and zero votes is invalid.
BFS returns the shortest path in operation count, but it returns nothing when the full goal is unreachable. Dijkstra explores all reachable states. When the goal is unreachable, it returns the path to the least misconfigured state. It fixes what it can and leaves the rest to an operator. The edge weight equals state misconfiguration times estimated operation time. Relative costs are 3 for a config update, 19 for an oplog update, and 101 for a node rebuild.
Each rule is a function of the current state and the SOT that returns an integer. Rules cover down node, voting, priority, oplog resize, hidden status, and availability zone imbalance. A curried closure, GetMisconfigFunc, binds the SOT once and sums the rules fast. Logs show each rule's part at each step.
Remediation steps are Temporal activities that do real I/O, so a search over them would take minutes per path. CommonContext offers ExecuteActivity, ExecuteLocalActivity, ExecuteWorkflow, and WithExternalDeps. In production it calls Temporal. In simulation, WithExternalDeps injects simulated providers with in-memory shard state. Operations become pure state changes, and hundreds of transitions run per second.
| No. | Chose | Instead of | Why |
|---|---|---|---|
| D1 | Search over operation sequences | Static ordered plugins and greedy single fixes | Plugins had circular dependencies, missed multiple failures, and tied safety checks to each layout. |
| D2 | Use Dijkstra with weighted edges | BFS only | BFS fails when no full healthy path exists, and Dijkstra gives partial remediation. |
| D3 | Run the real business logic in simulation | A separate planner | One code path stops the planner and the executor from drifting apart. |
| Shard | One slice of a database that holds part of the data. |
|---|---|
| Quorum | The majority of voting nodes needed to accept writes. |
| Oplog | The MongoDB log of writes that replicas copy. |
| Temporal | A workflow engine that keeps long tasks running through crashes and deploys. |