Tabletop exercise: a CAD outage at a 911 center

Reference article · 5 min read

A computer-aided dispatch outage is one of the few failures in a 911 center where the work does not stop. Calls keep arriving and the floor keeps dispatching with whatever still works. That makes CAD a useful subject for a tabletop exercise, because the hardest decisions are not technical. They are decisions about when to declare, who may move dispatch elsewhere, and how the center rebuilds the record afterward.

What follows is a scenario, the decision points worth arguing about, and a way to run the exercise without pulling anyone off the console.

Building the scenario and the injects

Set the exercise mid-shift, at a busy hour, with no confirmed cause.

The opening inject. CAD clients stop refreshing and unit status goes stale. Some workstations error, others hang. Radio and call handling still work, so calls keep coming. Say whether mapping, mobile data, and station alerting failed too.
Ambiguity on purpose. No ransom note, no security alert, no obvious hardware failure. Do not tell the room whether this is a malfunction or an intrusion.
Time pressure. Add a working structure fire, a pursuit crossing jurisdictions, and a repeat caller whose history cannot be retrieved.
A partial recovery. CAD returns at some positions, then fails again, testing whether the room reverses a continuity decision too early.

Decision point one: failure or intrusion

The center does not need attribution to keep dispatching, but the working assumption drives what IT does in the first hour. Rebooting servers, reimaging workstations, or restoring from backup can destroy data an investigation would need, so decide in advance who holds that call and what is preserved first. Many agencies treat an unexplained outage as a possible security incident until IT can say otherwise, then relax on evidence rather than impatience. Plan for two consequences: if the network may be involved, email and chat on it are poor coordination channels, and isolating a segment may take down something the floor needs.

Declaring continuity of operations and moving dispatch

Set the trigger in advance so nobody invents one under load. A threshold in elapsed minutes, or tied to a specific loss such as no unit status at any position, applies more cleanly under pressure than a judgment call.

Named authority. Identify the role that declares, plus alternates, around the clock. One person unreachable at 0300 is not a plan.
Cards and radio. Exercise the real card stock. Reserve a block of incident numbers for manual use so they do not collide when CAD returns.
Mutual aid transfer. Confirm the agreement states which call types transfer and for how long, and that the steps to reroute 911 trunks through the selective router or ESInet are documented with your provider. Ask what capacity the partner has while covering its own calls.

Reconstructing the call record afterward

Assign a scribe at declaration. Reconstruction is far cheaper when someone logs decisions and times as they happen.

Card discipline. Each card carries time received, location, nature, units, time stamps, disposition, and the reserved incident number.
Honest backfill. Enter manual incidents as after-the-fact entries that are visibly flagged, with real event times in the fields meant for them. Do not silently backdate. These records resurface in response time reporting, NFIRS and NEMSIS submissions, and records requests.
Cross-checks. Reconcile cards against recorded lines, radio logs, ANI and ALI data, and mobile data logs.
Retention. Retention and disclosure rules for 911 records vary by state and change over time. Confirm yours with the state archives authority and counsel.

Notifications and who makes them

Write the notification list before the exercise, then use the exercise to learn whether anyone knows it.

Local and state. Center manager, agency heads, the county emergency manager, the EOC if activated, and the state 911 program. Many states expect notification of PSAP service interruptions, so confirm your state's threshold, form, and timeframe.
Providers. Notify your 911 service provider promptly. Federal outage reporting duties generally fall on carriers rather than PSAPs.
Criminal justice data. If CAD touches CJIS data or shares infrastructure with systems that do, security incidents are reportable through your CJIS Systems Agency and state information security officer under the CJIS Security Policy. Confirm your state's current process.
Cyber channels. State CISO or fusion center, MS-ISAC, CISA, and the local FBI field office. Which reports are mandatory rather than voluntary keeps changing through federal rulemaking, so confirm obligations with counsel.

Running the exercise without taking dispatchers off the floor

Keep it discussion-based. A tabletop is a facilitated conversation, distinct from the drills and functional exercises described in federal exercise guidance.

Run short modules during shift overlap or in-service training, repeating the scenario for each shift rather than assembling everyone at once.
Seat supervisors, off-rotation telecommunicators, IT and the CAD administrator, radio staff, agency representatives, and a mutual aid partner.
Separately, test the manual process for a short window at a low-volume hour.
Close with a written improvement plan listing findings, owners, and due dates.

What to ask your CAD vendor and require in the contract

Ask what a workstation does when it loses the database or the network, and whether local caching exists. Ask for recovery objectives that match your deployed architecture, whether bulk after-the-fact entry is supported and audited, who answers a 24/7 call, and whether the vendor will support a forensic hold rather than reimaging.

In the contract, look for notification to the agency within a defined period when the vendor detects an incident affecting your instance, cooperation with an agency-led investigation, access to and preservation of logs, data ownership with export in a usable format, and security requirements that flow down to subcontractors and hosting providers.

HQT works with state and local agencies on public safety software, systems engineering, and cybersecurity, including continuity planning for dispatch.

Frequently asked questions

How long should a 911 center wait before declaring a CAD outage and going to manual dispatch?

Set the threshold in advance rather than deciding in the moment. Many centers use an elapsed-time trigger combined with a functional trigger, such as no reliable unit status at any position, so a supervisor can declare without debating the cause. Name the role that declares and its alternates for every hour of the day, and write the trigger into the continuity plan so it survives staff turnover.

Do we have to report a CAD outage to the state or to federal agencies?

It depends on your state and on what the outage turns out to be. Many states expect notification to the state 911 program when PSAP service is interrupted, and if CAD touches criminal justice information, security incidents are reportable through your CJIS Systems Agency and state information security officer under the CJIS Security Policy. Federal outage reporting obligations generally fall on communications providers rather than on the PSAP. Requirements change, so confirm current thresholds and timeframes with your state 911 program, your CJIS contacts, and counsel.

Should we treat an unexplained CAD outage as a cyber incident from the start?

A common approach is to assume it may be a security incident until IT can show otherwise, then step down as evidence comes in. The reason is practical: rebooting, reimaging, or restoring from backup can erase the data an investigation would need. Decide in advance who has authority to authorize restoration and what must be preserved first, and agree on a communication channel that does not depend on the possibly affected network.

How do we rebuild CAD records for calls handled on paper during an outage?

Assign a scribe at the moment of declaration, use cards that capture time received, location, nature, units, time stamps, and disposition, and reserve a block of incident numbers for manual use so numbers do not collide on restore. Enter those incidents into CAD afterward as clearly flagged after-the-fact entries with the real event times recorded, never as silent backdating. Reconcile the cards against recorded lines, radio logs, ANI and ALI data, and mobile data logs before the records feed reporting or a public records request.

Can we run a CAD outage exercise without pulling dispatchers off the floor?

Yes. A tabletop is a discussion-based exercise, not an operational drill, so it can run in short modules during shift overlap or scheduled in-service training with off-rotation staff. Repeat the same scenario for each shift instead of assembling everyone at once, and keep any hands-on test of the manual process to a brief window at a low-volume hour with supervisors present.