Running a disaster recovery failover test without breaking production

Reference article · 5 min read

A disaster recovery plan that has never been exercised is a document, not a capability. What stops most agencies from testing is a reasonable fear: that a failover will take down a permitting portal, a dispatch interface, or a payroll run, and that a self inflicted outage is harder to explain than a storm.

That fear is justified when the only test on offer is an all or nothing cutover of production. It is not justified when testing is treated as a ladder: each rung costs more, proves more, and is attempted only after the one below it stops producing surprises. A tabletop finds a missing after hours phone number for the cost of a conference room. A live failover finds it at two in the morning.

The four rungs and what each one proves

• Tabletop exercise. A facilitated walkthrough of a declared scenario against the written plan. It proves the plan exists, that people can find it, and that the authority to declare a disaster has a named owner and a backup. It proves nothing about the technology. Cost is the lowest on the ladder: facilitator preparation plus one block of staff time.
• Component failover. One dependency is failed over deliberately: a database replica promoted, a firewall pair cut to its partner. It proves the mechanism works and that monitoring notices, but not that the full stack returns in the right order. Cost is moderate, sits with one team, and is mostly preparation.
• Isolated bubble test. The recovery environment comes up on a segment with no route to production, using restored copies of real data. Systems start in dependency order and staff run real transactions. This is the highest value rung for most agencies: it measures real recovery time and data loss while production keeps running. It surfaces what only appears at boot, including hardcoded addresses, certificates never issued for the recovery site, licenses bound to hostnames, and batch jobs that wake up and reach for a live payment processor. Most of the staff hours go into building the segment and scoping the data.
• Full live failover. Production runs from the recovery site for a defined period, then fails back. It is the only test that proves the whole path, including public DNS, external interfaces, partner connections, and failback, which is often harder than the failover. It carries real outage risk and should never be a system's first live exercise.

Scoping one to a weekend maintenance window

Pick one system and its dependencies, not the portfolio. Build the schedule backward from the hour service must be normal, and keep the second half of the window for failback and verification.

1. Write success criteria before setting a date: the recovery time and recovery point you intend to demonstrate, and the transactions that must succeed for the test to count.
2. List every external dependency: identity provider, payment processor, state and federal interfaces, records exchanges, notification services. Mark each one in scope, stubbed, or blocked.
3. Walk the runbook on paper with the people who will execute it, timing each step honestly.
4. Publish that timeline with named owners, a go or no go checkpoint before cutover, and a hard failback deadline.

Safeguards agencies skip

• Freeze changes. No patches, schema changes, firmware, or configuration edits across the in scope systems before and after the test. Against a moving baseline you cannot tell whether a failure came from the recovery design or from last week's patch.
• Notify dispatch, the help desk, and the public information officer. Public safety centers and any agency sharing an interface need advance notice. The help desk needs a script so unrelated tickets are not blamed on the test, and the information officer needs prepared language before the window opens.
• Protect the source of truth. Confirm the restored data is a copy and that one independent, verified backup stays untouched and unused by the test.
• Document the abort point. Define in writing, before the window, what ends the test: a clock time, a set of symptoms, and one named person with authority to call it. Abort criteria decided in the moment are decided too late.
• Assign a scribe, and keep a channel out of band. One person records timestamps and observations and does nothing else. If the test touches email or phones, the team needs a channel that does not depend on what is failing over.

Turn findings into a corrective action list

Debrief while details are fresh, separating what happened from who did it. Every finding becomes a line item with a description, a severity, a named owner, and a due date, tracked where the agency already tracks work, not in a report that closes itself. Items that will not be fixed become accepted risks with a signature and a review date. Verify closed items before the next exercise. Repeating a finding across two exercises is a governance problem, not a technical one.

Continuity guidance and common control frameworks, including contingency planning material published by the National Institute of Standards and Technology, generally expect periodic, documented exercises. Obligations vary by program and data type, so confirm specifics with the governing authority or your counsel.

What to ask vendors and put in the contract

• Ask who performs the failover for a hosted system, how the agency requests a test, whether tests are billed, and what the most recent exercise measured, including corrective actions still open.
• Require in the contract a recovery time and recovery point objective with a stated method of measurement, the right to observe or join a periodic test, delivery of after action documentation, and notice before the provider changes recovery architecture or hosting region.

HQT provides cloud and enterprise infrastructure services for state and local government, including recovery design, runbook development, and exercise planning.

Frequently asked questions