EmergencyReparentShard operations are being hardened in upcoming release v25. In this blog, we cover how ERS works and the upcoming changes that make recovery safer, faster and less brittle

What is EmergencyReparentShard? #

EmergencyReparentShard (ERS) is the Vitess failover process used when a shard's current primary is dead or unreachable. While PlannedReparentShard gets a clean handoff from a healthy primary, ERS has to pick a replacement using only surviving tablets. It compares their transaction histories, promotes an eligible replacement, updates the topology and points the other tablets at the new primary. VTOrc uses ERS to resolve many unplanned failures automatically

The goal is to promote a tablet that has applied the most-advanced surviving transaction history as quickly as possible, as an outage of the primary blocks shard writes - this is an emergency!

A shard with 4 x tablets might look like this before ERS runs; notice the unavailable PRIMARY:

And after ERS:

At a high level, ERS moves through these phases:

1. Lock the shard. This prevents competing reparent operations from changing the shard at the same time
2. Stop replication receivers and collect state. ERS freezes the incoming transaction histories and checks what each reachable tablet has received and applied, giving it a stable view of the surviving data
3. Relaylog apply phase and history validation. The relaylog apply phase waits for received transactions to be applied from relay logs. ERS also checks for errant or conflicting histories so that primary selection is based on a history it can safely preserve
4. Choose a promotion candidate. ERS considers promotion rules, cell restrictions and durability requirements. The most-advanced tablet is not necessarily the final primary; if a different tablet is selected, it must first catch up from that source
5. Complete the reparent. ERS repoints replicas, ensures any required semi-sync acknowledgers are ready before promotion, and records the new primary in topology so the shard can resume writes

We need to be certain about the history we preserve, but every additional wait gives the ERS another place to fail. This matters for both manual reparents and automatic recoveries triggered by VTOrc

Optimizing relaylog apply phase using MySQL GTIDs #

TL;DR: before v25, lagging tablets unable to lead the election could still time out ERS. ERS now filters the relaylog apply phase by received GTIDs and races relay-log apply on tablets sharing the leading history, leading to faster emergency reparents that are less brittle

The Problem #

Comparisons of candidates in ERS consider 2 x MySQL replication positions: what a replica has received and what it has applied (the latter added to candidate sorting in v23 PR: #18531). Transactions can already be in its relay logs while the SQL thread is still working through them. Before promoting a replica, ERS must ensure it has applied everything it received

Before Vitess 25, the relaylog apply phase waited for every surviving tablet still under consideration to apply its relay logs. If any one of them exceeded --wait-replicas-timeout (defaults to 15s in vtctldclient and 30s in VTOrc), the entire ERS failed. This risk increases with the number of tablets in the shard: every additional tablet under consideration is another wait that can time out the reparent

The problem was that this included tablets we already knew were behind. Waiting for the eventual primary is necessary; letting a tablet that cannot lead the election fail the entire operation is not

Although this problem has affected Vitess users since ERS was introduced, it was first formally reported in issue #18529 around the Vitess 22 release in 2025. The issue described a shard with 4 x tablets: the primary and 2 x replicas were current, while another replica had substantial replication lag

This is not an unusual state. Network or I/O-thread delays, a busy or stopped SQL thread, or catching up after a restore can all leave a replica behind. Before v25, ERS still issued relay-log apply waits for these tablets even when their received histories meant they could not win; an apply backlog could turn an unnecessary wait into a timeout. For an automated VTOrc recovery, that meant retries or manual intervention while writes remained blocked

The Fix #

MySQL GTIDs give ERS a shard-wide view of how advanced each surviving tablet is. Close to the start of the operation, ERS stops the replication receivers and collects each reachable tablet's received and applied positions. The received history is now frozen; the SQL thread can keep applying it, but no new transactions arrive from the old primary

This distinction is important. If one tablet received transactions through 120 and another only received through 95 from the same history, waiting for the second tablet to apply through 95 cannot put it ahead of the first. Before v25, ERS already had this information but did not use it to narrow the relaylog apply phase

PR #20578 uses these frozen positions to identify the leading group early in the operation, before waiting for relay logs to apply

Improved v25 ERS:

1. Stops replication receivers and collects each surviving tablet's received and applied positions
2. Filters the relaylog apply phase to the most-advanced received histories
3. When those histories are equal, races relay-log application and continues as soon as the first tablet finishes applying
4. Completes the safety checks and primary selection, catching up a different promotion candidate if needed
5. Promotes the selected tablet and repoints the remaining tablets as part of the reparent

The example below uses transaction numbers from one shared history instead of full GTID sets. The MySQL lag values are illustrative, not derived from the transaction counts. Here, R2 wins the apply race and also satisfies the final promotion requirements:

Before v25, R3 could time out the entire ERS while applying its own received history (95), not while catching up to its peers (120). In v25, R3 is skipped during the initial relaylog apply phase

Why is this safe? R1 and R2 received the same most-advanced transactions, so applying their relay logs brings them to the same state. ERS needs one leading candidate to apply successfully, so a stalled or failing peer need not block the race. If no leading candidate completes, including when the sole leading candidate fails, ERS fails. After a successful apply, the relay log apply waits on non-race-winners are cancelled, but those tablets continue to apply. Positions are usually a good guide to which tablet will finish applying first, but slower hardware or competing workloads can change that. Racing the leading group lets reality prove who applies fastest, shortening the wait and helping ERS finish sooner

Winning that race is not an unconditional promotion. The most-advanced tablet can act as an intermediate replication source if the promotion rules or an explicit --new-primary request require a different primary. That candidate must catch up before it is promoted. The benefit is that ERS can move on without waiting for every peer to finish the relaylog apply phase

The existing promises and safety-checks of ERS are unchanged. Unreachable tablets are not automatically ignored: reachability and durability checks can still block ERS. Under semi_sync durability, an unreachable primary and another unreachable potential acknowledger can block ERS because the pair could still accept writes. This protection predates v25. Promotion rules, cross-cell restrictions, errant-GTID detection, semi-sync forward progress and shard-lock checks still apply. A tablet that returns an apply error is excluded from promotion and cannot count as a semi-sync acknowledger, but its received position is retained as evidence for errant-GTID detection

The relay-log waits share the configured timeout budget, including any additional waits needed after errant-GTID detection. A slow tablet can still delay another part of the reparent

This optimization depends on the received-history information available with MySQL GTIDs. File-position replication and MariaDB retain the existing wait-for-all behaviour on this path. For eligible shards there is no new flag to enable; this is the default in Vitess 25

Making position ordering more predictable #

TL;DR: candidate sorting could produce inconsistent results when GTID histories diverged. Since Vitess 23.0.6 and 24.0.3, ERS and PlannedReparentShard use consistent ordering that keeps a candidate behind any tablet with a strictly more complete history. These fixes are also included in v25

The Problem #

While improving candidate selection, there was another problem to address: GTID sets do not always have a simple ahead-or-behind relationship

When candidate histories form a simple ahead-or-behind chain, as with ordinary replication lag, the old sorter already worked. This bug matters when some histories are incomparable, for example after a split brain or an errant write on a replica. A divergent candidate could disrupt the ordering of otherwise comparable candidates, so the problem was not limited to choosing between the divergent histories

A simple example, using p, a and c as short labels for distinct originating server UUIDs:

• A has p:1-2,a:1-2
• B has p:1-2,a:1
• C has p:1-2,c:1

A is ahead of B because it contains all of B's transactions, plus a:2. C is incomparable with both: it has c:1, which they do not, and lacks their transactions from UUID a. GTID sequence numbers are per UUID, so c:1 is a transaction from a different origin, not a gap in the shared p:1-2 history. The old sorter compared candidates pairwise and treated incomparable histories as tied. Depending on map iteration or RPC completion order, C could disrupt the sort and leave B ahead of A, even though A has the more complete history

The Fix #

PR #20728 fixes this by counting how many other candidates strictly dominate each candidate's history. A candidate cannot rank ahead of a tablet that dominates it. Existing preferences, such as promotion rules, then break ties. This fix and its nil-alias follow-up (PR #20762) shipped in Vitess 23.0.6 and 24.0.3, and are included in v25

Using the same GTID sets:

• A has p:1-2,a:1-2 and is dominated by 0 candidates
• B has p:1-2,a:1 and is dominated by 1 candidate: A
• C has p:1-2,c:1 and is dominated by 0 candidates

Sorting reliably places A and C (0) before B (1). C's incomparable history can no longer cause B to rank ahead of A

ERS and PlannedReparentShard share this sorter, so both benefit from the fix. This makes the ordering consistent, but it cannot decide which side of an unresolved split brain to preserve. The next improvement gives operators an explicit way to make that choice

Strict recovery from split brain #

TL;DR: in v25, ERS on MySQL and Percona GTID shards tracks divergent leaders before filtering, prevents fallback to an older candidate if all leaders are removed, and adds explicit operator-controlled recovery. Choosing a history can discard transactions unique to the other branches; VTOrc never enables this override automatically

The Problem #

In a split brain, 2 x surviving tablets can each contain transactions the other lacks. Neither has the complete surviving history, so promoting either side means giving up transactions unique to the other

When ERS cannot determine a safe history automatically, the operator needs an explicit way to choose which side to keep. Falling back to an older replica is not a safe compromise: it can lose transactions from both sides

The Fix #

PR #20780 closes this gap and adds explicit split-brain recovery for MySQL and Percona GTID shards in Vitess 25. ERS records the divergent leaders before the relaylog apply phase and errant-GTID filtering. The default path can only proceed if that filtering leaves exactly one of the original leaders; otherwise ERS fails with the aliases and positions of the competing leaders

An operator who has determined which history to preserve can choose it explicitly:

vtctldclient EmergencyReparentShard <keyspace/shard> \
  --new-primary <tablet-alias> \
  --allow-split-brain-promotion

The flag is available only to shards using MySQL or Percona GTIDs. MariaDB and file-position replication remain on the existing non-GTID path and cannot use this override. The flag requires --new-primary, and the requested tablet must be one of the original undominated leaders (no other candidate contains a strictly more complete version of its history). ERS promotes exactly that tablet and preserves its full history

This is lossy recovery, not a merge. Transactions unique to the losing side will not be part of the new primary's history, and tablets from the losing side should be rebuilt. VTOrc never enables this automatically; choosing which data to preserve is an operator decision. Completed override promotions increment EmergencyReparentSplitBrainOverrides{Keyspace,Shard}, allowing operators to monitor and alert on its use

The override does not bypass the other promotion checks. The chosen tablet still has to apply its relay logs, satisfy promotion and cross-cell rules, make forward progress under the durability policy and pass the shard-lock checks. Only the chosen leader is waited on, so a losing branch stuck applying relay logs cannot block that wait

Summary #

ERS is one of the most critical operations in Vitess: it must resurrect a primary in the face of unplanned failure; every delay is more outage. In some common scenarios, Vitess 25 makes ERS safer, faster and less brittle. The biggest benefit is in environments where MySQL replication lag on some tablets would otherwise time out an ERS, despite an up-to-date replacement being available. Narrowing the relaylog apply phase and racing tablets with the same leading history can shorten recovery and let it succeed where it previously failed

Candidate ordering is now consistent. For MySQL and Percona GTID shards, split-brain checks now retain the original divergent leaders, preventing an older candidate from being promoted when errant-GTID filtering removes all of them. Operators also have an explicit recovery path to choose a leading history, accepting the loss of transactions unique to other branches. The other promotion checks still apply

All changes discussed here will be available in Vitess 25, expected in October 2026. See the in-progress Vitess 25 release summary and reparenting documentation for more detail

Further ERS improvements may be covered in future blog posts as more changes land

Links #

• PR #18531: EmergencyReparentShard: include SQL thread position in most-advanced candidate selection (Vitess 23)
• PR #20578: EmergencyReparentShard: only wait on relay-log apply for candidates that can win the election
• PR #20728: reparentutil: order reparent candidates by GTID dominance for a consistent sort (backported to 23.0.6 / 24.0.3)
• PR #20762: reparentutil: keep nil-alias tablets out of candidate ordering (backported to 23.0.6 / 24.0.3)
• PR #20780: EmergencyReparentShard: add explicit split-brain recovery
• PR #20831: EmergencyReparentShard: skip zero-position candidates in errant GTID detection