Tabletop exercise: a SCADA compromise at a water utility

Reference article · 6 min read

A tabletop exercise is a facilitated discussion, not a test. Nobody touches a control system, nothing is scored, and the purpose is to find the distance between what your emergency response plan says and what your staff would actually do at two in the morning on a holiday weekend. For a drinking water or wastewater utility, the most useful scenario is rarely the dramatic one. It is a slow, ambiguous compromise of the control environment, because ambiguity is what stalls real decisions.

The scenario below is built for a two-hour session with operations staff and the management chain in the same room. It assumes a small to mid-size system with a SCADA network, an HMI in the control room, a process historian, and at least one remote site on radio or cellular telemetry. Change the plant details so the room recognizes its own facility. If you want a prebuilt starting point, CISA publishes tabletop exercise packages at no cost, and EPA and WaterISAC publish water sector incident response guidance you can borrow structure from.

Set the scope and the ground rules

State the boundaries out loud before the first inject. Three ground rules do most of the work:

No fault-finding. Participants describe what they would do, not what they should have already done. Gaps are the output, not embarrassment.
Answer from current capability. If the procedure exists only in someone's head, say so. That is the finding.
Assume nothing is restored. Participants may not resolve the scenario by saying the vendor will fix it. Ask how long that would take and who makes the call in the meantime.

Assign a scribe whose only job is capturing decisions, unanswered questions, and names of people who were not in the room but should have been.

Roles to seat

Seat people by function, not by title. Missing seats are themselves a finding.

Facilitator and scribe. Neither participates in decisions.
Lead operator and shift operator. The people who would see it first.
Plant superintendent or operations manager.
SCADA or instrumentation technician. Often the only person who knows what is actually connected.
IT manager or contracted IT provider.
Water quality or compliance manager. Owns sampling, reporting, and the primacy agency relationship.
Utility director or general manager, with the city manager, county administrator, or board representative as an observer.
Public information officer and legal counsel.
Emergency management and, if practical, the systems integrator who maintains the control system.

The scenario and its injects

0:00 to 0:10. Framing. Scope, ground rules, and the state of the plant: normal flow, normal staffing, one operator on shift.

0:10 to 0:30. Inject one. The operator notices a chemical feed setpoint that does not match the shift log. Nobody on shift changed it. Ask: what does the operator do in the first five minutes? Who gets called, and by what method? Is there a written procedure for verifying a setpoint against a physical instrument? How would you tell a cyber event from a failing sensor or a bad edit?

0:30 to 0:55. Inject two. The historian shows trend data that contradicts field readings, and a second remote site reports values that are physically implausible. Ask: which source of truth do you believe now? Who is authorized to place the process in manual, and at what threshold? Who starts grab sampling, and does the lab have capacity? At what point does this stop being an operations issue and become an incident?

0:55 to 1:05. Break.

1:05 to 1:30. Inject three. The HMI locks. The operator cannot log in, and a message appears demanding payment. Ask: can you run the plant blind? For how long, with how many people, and do written manual operating procedures exist? Do you isolate the control network, and who physically does it? Do you preserve evidence before restoring, and who decides? Where are the PLC program backups, when were they last verified, and can anyone load them?

1:30 to 1:45. Inject four. A local reporter calls and a council member emails asking whether the water is safe. Ask: who speaks publicly, what is the holding statement, and what would trigger a public notice or advisory, which is issued in coordination with the state primacy agency?

1:45 to 2:00. Hotwash. Each participant names one thing that surprised them and one thing they will change.

The decision points that matter

Three decisions carry the exercise. Do not let the room skip past them.

When to go to manual. The trigger should be written and conservative. Treat loss of confidence in the control system as loss of the control system.
Whether treatment continues. Continuing with degraded visibility, reducing production, or stopping and drawing on storage are all defensible. The room should know who has authority to choose, after hours and on a weekend, and what the water quality monitoring plan looks like in each case.
Who notifies whom, and when. Your state drinking water primacy agency, your regional CISA cybersecurity advisor and CISA's 24/7 reporting line, the FBI field office, local emergency management, your insurer, and WaterISAC if you are a member. Federal incident reporting obligations for critical infrastructure have been the subject of ongoing rulemaking, and requirements differ by sector and system size, so confirm what applies to you with the official source and your counsel rather than assuming.

After-action items these exercises reliably surface

No current control system asset inventory or network diagram.
Unknown or unmanaged remote access paths: vendor VPNs, cellular modems, and dial-in lines nobody owns.
Manual operating procedures that exist as institutional memory rather than documents.
PLC and HMI backups that have never been test-restored.
No out-of-band contact list, because the call tree lives on the network.
Undefined notification thresholds and unclear after-hours authority.
No mutual aid arrangement with a neighboring utility for operators or equipment.

Assign every item an owner and a date, and re-exercise the same scenario after the fixes land.

What to require of your integrator and support vendors

Exercises usually expose contract gaps. In the next service agreement or solicitation, ask how remote access is authenticated and logged, who holds the current PLC logic and configuration backups and how you obtain them, what the response commitment is for a suspected compromise, and what the vendor's own breach notification obligation to you is. Require named personnel, defined response times, and participation in at least one exercise per term.

HQT works with state and local agencies on cybersecurity and systems engineering for operational technology environments.

Frequently asked questions

How long should a water utility cybersecurity tabletop exercise be?

Two hours is a practical length for a first exercise and fits within a single shift change or a regular staff meeting block. That allows roughly ten minutes of framing, three or four injects with twenty to twenty-five minutes of discussion each, a short break, and a fifteen-minute hotwash. Longer sessions tend to lose operations staff who still have a plant to run. If you need more depth, run a second session on a narrower question such as manual operations or notification.

Who do we notify if we suspect a cyberattack on our SCADA system?

Typical notifications include your state drinking water primacy agency, your regional CISA cybersecurity advisor and CISA's central reporting line, your local FBI field office, local emergency management, your insurer, and WaterISAC if your utility is a member. Reporting obligations vary by sector, system size, and state, and federal critical infrastructure incident reporting rules have been developing over time. Build the contact list and the thresholds into your emergency response plan in advance, and confirm what legally applies to your system with the official source and your counsel.

Does federal law require water systems to run tabletop exercises?

Federal law directs community water systems above a population threshold to conduct risk and resilience assessments and to prepare or update emergency response plans, with certifications to EPA on a recurring cycle. Exercising those plans is strongly encouraged by EPA and by sector guidance, and some states impose their own expectations. Because thresholds, deadlines, and state requirements change, confirm your specific obligations with EPA, your state primacy agency, or your counsel rather than relying on a general summary.

Should we shut down treatment if we think the control system is compromised?

There is no universal answer, which is exactly why the decision belongs in a tabletop exercise rather than in the middle of an incident. The realistic options are continuing under manual operation with increased bench sampling, reducing production and drawing on storage, or stopping treatment entirely. What matters is that your plan names who has authority to choose after hours, what water quality monitoring applies in each case, and when the state primacy agency is brought into the decision.

Do we need IT staff in the room, or is this an operations exercise?

Both. Operators see the first symptoms and run the plant in manual, while IT or your contracted IT provider handles isolation, evidence preservation, and coordination with outside responders. Exercises frequently reveal that neither group knows what the other assumes about the control network, including which remote access paths exist. Seating the SCADA or instrumentation technician alongside IT is usually where the most valuable findings come from.