Replica corroboration
Each replica reports its view through the WAL receiver status. The primary is unreachable and a replica still streams from it: no failover. Everything is unreachable: Admin waits, because there is no evidence and no candidate.
Each Neki shard has one primary and several replicas. A failed health check can mean a dead primary, or a network split that isolates only the control plane. Promoting on that signal alone risks two writable primaries.
Admin runs health checks and decides on failover. PostgresManager controls each Postgres process and reports exits. When the check on primary P fails, Admin queries the replicas for their replication state. If a replica still streams from P, Admin does nothing.
If a replica reports broken replication and none disagree, P counts as dead. Admin then fences P by disconnecting the reachable replicas. It picks the best replica, creates replication slots on it, and promotes it.
It points the routers at the new primary and repoints the other replicas. Finally it marks P as TEARDOWN.
Each replica reports its view through the WAL receiver status. The primary is unreachable and a replica still streams from it: no failover. Everything is unreachable: Admin waits, because there is no evidence and no candidate.
Admin prefixes the primary hostname in each replica's connection string with two slashes. The replica can no longer find P, and the change is reversible. Postgres has no command to stop only the replication IO thread, so this stands in for it. A returning P cannot collect acknowledgements under ANY 1, so it cannot commit.
Admin finds the highest received WAL position among reachable replicas. It prefers a promotable replica that has replayed through that position, and waits briefly for it. If none does, it takes the promotable replica with the highest receive position, using replay position to break ties.
| No. | Chose | Instead of | Why |
|---|---|---|---|
| D1 | Require replica evidence before failover | Repeated health checks alone, or promoting on Admin's failed check | Retries do not help when a partition isolates Admin, and a wrong promotion creates two primaries. |
| D2 | Choose by highest received WAL | Choose by replay position | WAL is sequential, so a higher received position contains all earlier WAL. |
| D3 | Keep the old primary out after failover | Rejoin it as a replica | It may hold writes the new primary lacks, and logs and disk stay for postmortem. |
| WAL | The write-ahead log that Postgres replicates to replicas. |
|---|---|
| Fencing | Cutting off the old primary so it cannot accept writes. |
| Split brain | Two nodes both believe they are the primary. |
| Replication slot | A marker that keeps WAL for a replica until it reads it. |