Writing runbooks your staff will actually use at 2 a.m.

Reference article · 7 min read

At 2 a.m. the person answering the page is rarely the person who built the system. It is the newest analyst on rotation, or a network technician covering for someone on leave, working from a phone in a dark kitchen with a laptop balanced on the counter. Whatever that person can read, understand, and act on in the next ten minutes is the operational capability the agency actually has. Everything else is documentation that exists.

Runbooks are written for that moment. They are not summaries of how a system works, and they are not statements of policy. They are step-by-step instructions for one recurring task, written so that a competent person with no deep knowledge of the system can carry it out safely, confirm it worked, and undo it if it did not.

What a runbook is, and what it is not

Agencies often fold three different documents into one and end up with something that serves none of the three purposes well. Keep them separate and cross-reference between them.

• Policy answers who is permitted to do something and under what authority. It changes slowly, is approved at a management level, and belongs in the agency's governance repository.
• Training material builds understanding. It explains architecture, concepts, and the reasoning behind design choices. Someone reads it once, in daylight, before they are on call.
• A runbook covers one task, executed under time pressure, by someone who may be following it for the first time. It assumes the reader does not have time to learn the system tonight.
• Design and architecture documents describe how the system is put together. A runbook can name one, but it should never require the reader to consult it mid-procedure.

If a paragraph in your runbook explains why something is designed the way it is, move it into the training material and leave a single sentence of context behind.

What belongs in every runbook

A usable runbook entry has a predictable shape. Readers under stress rely on that predictability.

• Trigger. The specific condition that makes this the right procedure: an alert name, a ticket type, a symptom described the way the caller will describe it. State plainly when this runbook does not apply.
• Preconditions and authority. What access, credentials, and approvals the reader must have before starting, and who can grant them at night. If a change control exception is needed, say which one and who approves it.
• Impact statement. What breaks, and for how long, if the procedure runs. Whether users lose sessions. Whether the action is reversible.
• Exact steps. The actual commands, the actual menu path, the actual button label, with device hostnames as they appear in inventory. Not "fail the tunnel over" but the literal sequence, in order, numbered.
• Verification. How the reader knows it worked, stated as an observable result: this dashboard turns green, this test query returns rows, this endpoint answers. Include the expected wait before the check is meaningful.
• Rollback. The steps to return to the prior state, with the same level of precision, plus the point past which rollback is no longer possible.
• Escalation. A name, a role, and a number, with the hours each applies. Say when to escalate rather than continue, and make that threshold a time limit, not a judgment call.
• Record keeping. Which ticket to update, what to note, and who must be notified when the task is complete.

An example: failing a site over to the backup circuit

Consider a common county or state task, moving a remote office onto its secondary WAN or VPN circuit. The runbook entry would name the trigger as the specific alert or the branch office reporting loss of connectivity, and would rule out the case where the whole region is down, which points to a different procedure.

Preconditions would name the credential vault entry required, the console access path that does not depend on the tunnel being restored, and whether after-hours failover is pre-approved or requires a call. The impact statement would note that active sessions drop and that the backup path may not support the same throughput, so certain applications will be slow until failback.

The steps would name the exact devices, in order, with the exact commands or interface clicks. Verification would be concrete: the interface shows the expected state, a test from the branch reaches a named internal resource, and the monitoring check clears within a stated number of minutes. Rollback would cover returning to the primary once it is stable, including any manual step that does not happen automatically. Escalation would name the carrier ticket process, since at that hour the next action is often someone else's.

Write the real version with the real names. A generic runbook is a training document wearing the wrong label.

Where the runbook lives when the network does not

The most common failure is storing the recovery procedure inside the thing being recovered.

• Keep an offline copy that does not require the VPN, the identity provider, or the intranet to open. A synced local folder on every on-call laptop and phone is a reasonable baseline.
• Keep a printed set in the operations center and, if you have one, at the alternate facility. Refresh it on a schedule rather than when someone notices it is stale.
• Never put passwords or keys in the runbook. Point to the vault entry by name, and make sure the break-glass path to that vault is itself documented and tested.
• Control access. Runbooks describe exactly how to take systems down, so treat them as sensitive operational material, not as open agency content.

Keeping runbooks alive

Runbooks decay quietly. Firmware changes, a portal redesigns its menus, a phone number moves, and nobody finds out until the next outage.

1. Review after every use. The person who ran the procedure edits it within the same week, while the friction is still fresh. Make this an expected closeout step on the incident ticket, not a favor.
2. Give each runbook an owner and a review date. An unreviewed runbook past its date should be visibly flagged, and one nobody will own should be retired rather than left to mislead someone.
3. Test with the newest person. In a maintenance window, have the least experienced qualified staff member execute the runbook while the author watches in silence. Every question they have to ask out loud is a defect in the document. Fix it before the window closes.
4. Tie updates to change management. Any change that alters a documented step should list the affected runbook in the change record.

How to start, and what to require of vendors

Start from your own on-call history. Pull the tasks your staff actually performed after hours over the past several months, rank them by frequency and by how much damage a mistake would cause, and write the top handful. Ten good runbooks beat a hundred placeholders.

When a vendor operates or implements a system for you, put the expectations in writing:

• Runbooks for agency-performed operational tasks as a named deliverable, with acceptance based on agency staff executing them successfully, not on page count.
• Delivery in an editable format the agency owns and stores in its own repository, not only in a vendor portal.
• An obligation to update affected runbooks when the vendor makes a change, and to hand over current versions at the close or transition of the contract.
• Clarity on which tasks the agency may perform without affecting support terms, so staff are not reading a procedure they are contractually barred from running.

HQT works with state and local agencies on systems engineering and operational documentation, including runbook development and transition planning.

Frequently asked questions