Anurag’s Read

Designing a reliable unplanned failover.

A clear walk through detecting a dead Postgres primary without split brain, with exact rules for fencing and candidate choice. Anyone running replicated databases can reuse the ideas, and the admitted loss cases make it credible.
Score
9.0
Verdict
Must read
Blog and date
PlanetScale,
Original length and level
11 min, advanced
Topics
postgres, failover, replication, reliability
Read the original

Context

Each Neki shard has one primary and several replicas. A failed health check can mean a dead primary, or a network split that isolates only the control plane. Promoting on that signal alone risks two writable primaries.

Architecture

Admin runs health checks and decides on failover. PostgresManager controls each Postgres process and reports exits. When the check on primary P fails, Admin queries the replicas for their replication state. If a replica still streams from P, Admin does nothing.

If a replica reports broken replication and none disagree, P counts as dead. Admin then fences P by disconnecting the reachable replicas. It picks the best replica, creates replication slots on it, and promotes it.

It points the routers at the new primary and repoints the other replicas. Finally it marks P as TEARDOWN.

Mechanisms

M1

Replica corroboration

Each replica reports its view through the WAL receiver status. The primary is unreachable and a replica still streams from it: no failover. Everything is unreachable: Admin waits, because there is no evidence and no candidate.

M2

Fencing

Admin prefixes the primary hostname in each replica's connection string with two slashes. The replica can no longer find P, and the change is reversible. Postgres has no command to stop only the replication IO thread, so this stands in for it. A returning P cannot collect acknowledgements under ANY 1, so it cannot commit.

M3

Candidate choice

Admin finds the highest received WAL position among reachable replicas. It prefers a promotable replica that has replayed through that position, and waits briefly for it. If none does, it takes the promotable replica with the highest receive position, using replay position to break ties.

Design decisions

No.ChoseInstead ofWhy
D1Require replica evidence before failoverRepeated health checks alone, or promoting on Admin's failed checkRetries do not help when a partition isolates Admin, and a wrong promotion creates two primaries.
D2Choose by highest received WALChoose by replay positionWAL is sequential, so a higher received position contains all earlier WAL.
D3Keep the old primary out after failoverRejoin it as a replicaIt may hold writes the new primary lacks, and logs and disk stay for postmortem.

Key numbers

  • Default durability is ANY 1 over three replicas, with synchronous commit on.
  • Worked example positions: R1 120, R2 125, R3 123 received.
  • Detection covers crash, refused connection, dropped packets, and frozen process.

Failures and limits

  • ANY 1 means a commit reached some replica, not necessarily a surviving one.
  • If the primary and that replica both fail, the commit is lost.
  • Admin cannot promote a replica it cannot reach, and WAL held only on lost machines is gone.
  • A replica that was ahead but not promotable needs a rewind or rebuild.
  • The new primary's first commits wait for a replica to attach and acknowledge.

Lessons you can use

  • Never act on one observer's failed check. Ask independent witnesses first.
  • Fence before you promote, and make the fence reversible.
  • Choose the candidate by data position, not by health or name.
  • Document which failures lose data.

Glossary

WALThe write-ahead log that Postgres replicates to replicas.
FencingCutting off the old primary so it cannot accept writes.
Split brainTwo nodes both believe they are the primary.
Replication slotA marker that keeps WAL for a replica until it reads it.