Incident postmortems after an outage: blameless review and follow-through

Reference article · 6 min read

An outage ends when service is restored. The review that follows decides whether the same failure happens again. For a public agency that review carries extra weight: the document you write may be disclosable under your state's public records law, elected officials will ask what happened before you have finished finding out, and the system that failed is often operated by a vendor rather than by your own staff.

A postmortem is an operational document written for the people who will be on call next time. It is not a disciplinary file and not a press statement. Keeping it blameless is not politeness. It is the only practical way to get accurate information out of the engineer who ran the command, the help desk lead who waited an hour to escalate, and the vendor who would rather describe the event in general terms.

What blameless actually means

Blameless does not mean nobody is accountable. Corrective actions still get owners, and serious misconduct is still handled through the channels that handle misconduct. What blameless means is that the review asks why an action looked reasonable to the person taking it, using the information they had at that moment, rather than asking who should have known better.

• Keep the review separate from personnel process. Say so out loud at the start of the meeting, and mean it. One review used as the basis for discipline will end candid participation for years.
• Write roles, not names. "The on-call engineer" and "the network team" are enough. This also reduces the risk of a released document reading as an accusation against an identifiable employee.
• Treat confusion as a finding. If someone could not tell whether the alert was real, that is a defect in the alert, not in the person.
• Include the near misses. The decision that happened to work but could easily have made things worse is often the most useful item in the document.

Build the timeline before anyone builds a theory

Reconstruct the sequence from logs, ticket timestamps, chat history, and phone records before the group starts explaining. Establish at minimum: when the first adverse effect occurred, when monitoring first detected something, when a human first knew, when the incident was declared and escalated, when mitigation began, when service was restored, and when the backlog created by the outage was cleared.

Two intervals deserve their own line in the document. The detection delay is the gap between the first adverse effect and the first signal that anything was wrong. The escalation delay is the gap between someone knowing and the right people being engaged. These gaps are usually where the recoverable time is, and they are almost always fixable with changes to tooling and paging rules rather than changes to people.

Describe impact in service terms

Write impact as a resident or a caseworker experienced it, not as the infrastructure experienced it. Which services were unavailable or degraded. Who could not do what: file, pay, renew, look up, dispatch, receive a benefit determination. Whether anything was lost rather than delayed. Whether any statutory or regulatory deadline was affected, and what the agency did about it. Whether a manual workaround was used and what backlog it created.

This framing serves two purposes. It tells operations staff what to protect first next time, and it gives your public information officer and your program directors language that is accurate without being technical.

Contributing conditions, not a single root cause

Outages in real systems rarely have one cause. Replace "root cause" with a short list of contributing conditions: the triggering change or failure, the conditions that let it spread, the reasons detection was slow, and the reasons recovery took as long as it did. Expired knowledge, a runbook that pointed at a decommissioned tool, a maintenance window nobody outside IT knew about, and a monitoring gap can all appear in the same list.

Corrective actions then attach to conditions. Each action needs a named owner, a due date, and a place where it is tracked alongside ordinary work. Actions that live only in the postmortem document do not get done. Review open actions on a standing cadence and close the loop in writing, including actions the agency decides not to take and why.

Public records, elected officials, and the press

Assume the postmortem is a public record. Most state public records acts reach internal agency documents, and exemptions for security information, attorney work product, or ongoing investigation are narrow, state specific, and change over time. Confirm the treatment with your agency counsel and your records officer before the first incident, not during one.

Practical consequences: write the document so you would be comfortable seeing it quoted. Keep specific vulnerability details, credentials, network diagrams, and anything that would help an attacker in a separate technical annex, and ask counsel how that annex should be handled. If the incident involved suspected unauthorized access or personal information, involve counsel and your privacy officer immediately, because breach notification obligations run on their own timelines and vary by state and by data type.

For elected officials and the press, publish what is confirmed and label what is not yet known. Give the impact, the current status, and when the next update will come. Do not offer a cause while the timeline is still being assembled. A correction later costs more credibility than a short initial statement.

When the failed system belongs to a vendor

For a hosted or software as a service system, your agency still owns the impact and the communication, but not the evidence. You have to ask for it, and the time to establish that right is at contract signature.

• Ask candidate vendors how they conduct internal postmortems, who receives them, and whether customers see the full document or a summary.
• Require written incident notification within a defined period, a written postmortem within a defined period after restoration, and a named accountable contact during an incident.
• Require that the postmortem include a timeline with timestamps, the customer impact, the contributing conditions, and corrective actions with target dates, then require status on those actions until they close.
• Reserve the right to hold a joint review with your staff present, and make it a standing agenda item in vendor performance meetings.
• Keep credit or service level remedies separate from the review. Remedies are a contract matter; the review is how you decide whether to keep trusting the platform.

Getting the first one done

Start with a one page template and a standing rule that any unplanned outage above an agreed severity gets a review within a set number of business days while memories are fresh. Invite operations, the help desk, the program owner whose service failed, and the vendor where relevant. Assign a facilitator who did not work the incident. Publish the result internally, track the actions, and report closure. The discipline matters more than the format.

HQT works with state and local agencies on IT operations, systems engineering, and vendor oversight practices, including incident review and follow through.

Frequently asked questions