Your failover depends on the thing that failed

A financial services client ran a dependency audit on a payment system that had been signed off as regionally redundant. They found seventeen paths back to us-east-1.

None of them were in the architecture diagram. Most were not in anybody’s mental model either — a secrets manager endpoint hardcoded during a migration, a monitoring agent reporting to a single collector, a certificate authority, an internal DNS zone nobody had thought about since it was created.

The system was redundant in the sense that the compute and the database were replicated. It was not redundant in the sense that it would keep working.

Redundancy is a property of the whole path

The failure mode that catches people is not the one they designed for. Nobody is surprised when a region hosting their application goes down. They are surprised when the region is fine, the replica is fine, and the failover does not execute — because the thing that triggers the failover was itself in the failed region.

This shows up in a specific and repeatable set of places:

• Identity. If the control plane you authenticate against is regional, and it is down, nobody can log in to run the runbook.
• DNS and traffic management. Health checks and record updates have a control plane. Data planes usually survive; control planes are where the dependency hides.
• Observability. If your dashboards and alerting live in the affected region, you lose the ability to see what is happening at the moment you most need it.
• Deployment. If failover requires a pipeline run, and the pipeline is regional, the failover requires the region.

Each of these is a service people describe as global. Several of them are global in the sense that they present one endpoint, and regional in the sense that something behind it is not.

Audit the path, not the architecture

The diagram will not tell you this. Diagrams show intent. What you need is the set of network destinations and API endpoints the system actually touches during a failover, which is a different exercise and a considerably duller one.

The version that works is unglamorous: trace an actual failover, in a real environment, and write down everything it calls. Not a tabletop. Not an annual test with a scheduled window and everybody watching. A rehearsal where the answer is allowed to be no.

Seventeen dependencies is not an unusual number. It is what a system accumulates over four years of small expedient decisions, each of which was correct at the time and none of which was recorded as a resilience decision.

The uncomfortable version

Most organizations that believe they are multi-region are running an active-passive design whose passive side has never carried production traffic, and whose failover has never been executed under conditions the operator did not control.

That is not the same as having no plan. It is worth knowing which one you have before an outage tells you.