Specialist technology and software for state and local government info@hqt.orgCommunitiesGlossarySearch

Security

Run the tabletop for an outage, not just a breach

A tabletop built around a failed migration or a vendor outage tests decisions a ransomware scenario never reaches.

Ransomware is the scenario most continuity plans get exercised against, and for good reason: the consequences are severe and the story is easy to picture. But think back to the days when a counter could not take payments or staff went back to paper. The cause was often more ordinary. A migration cutover did not go as planned. A hosted vendor had an outage. A certificate expired. A storage system failed on a holiday weekend.

The organizational questions in those events are close to the ones a breach raises. Who decides to stop? What do residents get told, and by whom? How long can the office run on a manual process? An agency that exercises only the attack scenario is rehearsing part of its exposure. The format is already built, so the change is small: run the same tabletop with a different cause.

What an outage scenario surfaces that a breach scenario does not

A breach exercise converges on containment, legal notification, and insurance. Those matter, but they are specific to an adversary. An outage exercise pushes on a different set of questions.

  • Rollback. A migration scenario forces a decision that is easy to defer: the deadline for rolling back, the conditions that trigger it, and who has the authority to call it.
  • Dependence on someone else's clock. When a vendor is down, the agency does not control restoration. The exercise reveals whether anyone knows the escalation path or the contractual remedy.
  • Degraded service rather than stopped service. In a breach, systems are often taken offline deliberately. In an outage, one system is gone and the rest still work, which creates partial workflows staff need instructions for.
  • No adversary to point to. Communication has to explain a self-inflicted or vendor-caused problem, which is a harder message than reporting a criminal act.

Three scenarios worth adding

The failed cutover. A permitting or utility billing system was migrated over a weekend. Monday morning, data for a subset of accounts is wrong and staff notice at the counter. The vendor says a fix is possible but will not commit to a time. The rollback window closes at noon.

The vendor outage. A software-as-a-service platform the agency depends on is unavailable. The status page says the provider is investigating. It is payroll week, and there is still no restoration estimate.

The quiet dependency. An identity provider, a certificate, a DNS record, or a network path fails, and several unrelated applications stop working at once. Staff cannot log in to the tools they would normally use to coordinate a response.

Run it the way you run the breach exercise

The mechanics do not change. A facilitator, a scribe, a fixed time box, and the same people in the room: the administrator or manager, IT, finance, HR, the attorney, the public information officer, and the program managers who own the affected service. Deliver the scenario in phases, with new information at each one. Set a ground rule that nobody fixes anything during the session. The purpose is to find gaps, not to close them.

Published guidance covers the format. FEMA's Homeland Security Exercise and Evaluation Program describes exercise design, conduct, and evaluation, including the after-action process. NIST Special Publication 800-84 covers test, training, and exercise programs for IT plans, and Special Publication 800-34 covers contingency planning. CISA publishes tabletop exercise packages that can be adapted to a local scenario. The material is federal, and the structure transfers.

Questions that produce the most value

  1. What is the recovery time objective for this service, and who set it? If the answer came from IT rather than from the program owner, that is a finding.
  2. How long can the office operate manually, and is the manual process written down and current?
  3. Who is authorized to call for a rollback, and what happens when that person is unreachable?
  4. Where is the vendor contact information, and does anyone know which support level was actually purchased?
  5. Does a statutory or charter deadline fall inside the outage window? Payroll dates, tax deadlines, election calendars, and public records response clocks do not pause because a system is down.
  6. If the primary communication tool is the system that is down, what replaces it?

What we see in our own drills

We run failure drills against our own systems: pulling a dependency, expiring a credential in a test environment, exercising a restore. The same findings keep coming back, and they are worth checking in your own procedures.

Runbooks assume the monitoring system is available. Recovery procedures list steps but not decision points, so the team knows how to restore and not when to stop trying and fail over. Restore procedures are written down, but the elapsed time stays unknown until someone measures it. Contact lists go stale because they are written once and not maintained.

Where to start

Add one outage scenario to the exercise calendar this year. If an annual ransomware tabletop is already scheduled, make the next one a failed migration or a vendor outage rather than a repeat of the attack. Write an after-action report with a named owner and a date for each corrective action. Put the recovery time objectives in front of the program managers who should be setting them.

Get in touch

Have a project or a solicitation coming up?

Talk with an HQT engineer. We respond to agency, corporate, and partner inquiries promptly.