Why we write the runbook before we write the feature

June 10, 2026 · 5 min read · HQT

In our own work, the first document written for a new service is not a design document or a user story backlog. It is the runbook: the page that tells an operator what the service does, how to tell whether it is healthy, what to do when it is not, and who to call when the answer is not written down. We draft it before we build the feature it describes.

A runbook for a system that does not exist yet is partly fiction. That is why it is useful. Writing the operational story early forces design decisions into the open while they are cheap to change, rather than after the system holds real records.

What we mean by a runbook

A runbook is written for a person who did not build the system and is reading it under pressure. Ours cover:

• What the service is for, who owns it, and what depends on it
• How it is deployed, and how a deployment is rolled back
• The signals that indicate it is healthy, and where those signals are visible
• The failure modes we expect, with the steps for each
• Backup and restore, including how long a restore takes and what it produces
• What access an operator needs, and how emergency access is granted and recorded
• Routine maintenance: certificate renewal, key rotation, log and data retention

It is not a user manual or an architecture diagram; it is the operating instructions.

The first draft is a design review

The draft fails in specific places, and those failures are the useful part. If the recovery step for a stalled job is "restart it," someone has to confirm the job can run twice without duplicating work. If the fix for a bad record reads "edit the row in the database directly," the system is missing an administrative function that staff will eventually need. If recovery requires replaying work that was in flight, the queue has to be durable, which is cheaper to decide now than later.

The sentences that are hardest to write point at the weakest part of the design. A procedure nobody can describe plainly will not work at three in the morning.

Questions the runbook asks before the first feature branch

1. How does an operator learn the service is broken before a member of the public reports it?
2. What is the safe action when the cause is unknown?
3. What is lost if the service is down for an hour, and what is lost after a day?
4. Can any step here be performed by someone who is not a developer?
5. What does a restore produce, and has anyone watched one finish?
6. What is the rollback, and is it still available after a database schema change?

These are ordinary engineering questions. The difference is when they are asked. Adding a health endpoint, making a job idempotent, or building an administrative screen is routine work in the first week and disruptive work during the first outage.

Writing it last produces a different document

A runbook written after launch describes what the system happens to do, not what it should do. Requirements that would have changed the design are expensive by then, so they get recorded as warnings instead: do not restart this during business hours, do not run the import twice, ask a specific engineer before touching the queue. Each is the residue of a decision an early draft would have surfaced.

Operations is the long part of the lifecycle

Public sector systems often stay in service for many years, and the people running one late in its life are rarely the people who built it. NIST Special Publication 800-53, the security and privacy control catalog, includes contingency planning and incident response control families that call for plans to be documented, tested, and updated. The NIST Cybersecurity Framework covers similar ground in its Respond and Recover functions. Many state and local programs adopt or adapt them, though obligations vary by jurisdiction. A system only one person knows how to operate is a continuity problem regardless of how well it is coded.

Keeping the runbook honest

We keep runbooks in the same repository as the code and review them in the same pull request. A change that alters an operational procedure is not finished until the procedure changes with it. We also exercise them internally: restore a backup into a scratch environment and time it, then have someone who did not write it follow the document step by step without asking the author questions. Each gap it finds becomes a correction to the document or a change to the system. An untested runbook is an assumption.

What this means for an agency

Ask for the runbook as a named deliverable with acceptance criteria, not as an appendix at closeout. Ask when it was written: a draft that predates the code was doing design work, and one written the week before go-live was doing paperwork. Require a walkthrough in which agency staff, not the vendor, perform a restore and a rollback and record how long each takes. Keep the document in a repository the agency controls. Then check it on a schedule: a procedure nobody has followed since the system last changed is unverified.