Alert fatigue: tuning monitoring so real outages get noticed

Reference article · 6 min read

Every operations team that has run monitoring for a few years arrives at the same place. The tool sends more alerts than anyone can read, staff learn which ones are safe to ignore, and the habit of ignoring eventually covers the one alert that mattered. The outage is then reported by a clerk at a counter, not by the system bought to catch it.

Alert fatigue is a configuration problem before it is a staffing problem. Telling people to read every alert does not work once the volume is past reading. What works is reducing what the system is allowed to say, so that anything it does say is worth stopping for. None of what follows turns monitoring off. It moves most of it out of the notification path and into dashboards, where it belongs.

Where the noise actually comes from

Noise is rarely one bad rule. It is a handful of patterns that each look reasonable alone:

• Per-device alerting. A default template applied to every server, switch, and interface produces alerts in proportion to inventory rather than impact.
• Thresholds that flap. A CPU or latency threshold with no sustained duration fires and clears repeatedly during normal load, producing pairs of messages that carry no information.
• Duplicates from several tools. The network platform, the server agent, and a cloud provider's native monitoring each notice the same failure, and each notifies separately.
• Scheduled work reported as failure. Backup warnings, patch reboots, and batch jobs that retry successfully arrive overnight as alerts even though the expected outcome was reached.
• Conditions nobody will act on at night. A file server disk crossing a capacity threshold is a work item, not a 2 a.m. page. Sending it as a page teaches the on-call person that pages are optional.

Make every page carry an action

The most effective single rule is that any alert routed to a human at night must name something that human can do. Before a rule may page, three questions need written answers: what is broken from a user's point of view, what does the responder do first, and what happens if it waits until morning. If the third answer is "nothing much", it is not a page.

Sort every existing rule into three destinations: page a person now, open a ticket for business hours, or record to a dashboard with no notification. Most belong in the second and third groups. Put the first response step, the runbook location, and the affected service in the alert text itself, so the responder does not start from a device name.

Alert on symptoms, not on every component

Component alerting asks whether each piece is healthy. Symptom alerting asks whether the service that staff and the public use is working. A check that loads the permit portal, submits a test query, or confirms that dispatch is accepting records tells you more than dozens of node alerts, and it produces one notification instead of many when a shared dependency fails.

Keep the component data: collect it, chart it, retain it for capacity planning and root cause work, and stop letting it page. Component metrics are how a responder diagnoses a problem once a symptom check has said there is one.

Suppress what is already explained

• Dependency suppression. Define which services sit behind which links, hosts, and sites, so a failed upstream device suppresses the alerts it causes downstream. One cause, one notification.
• Sustained conditions only. Require a threshold to hold for a defined period before alerting, and require recovery to hold before clearing. This removes most flapping without hiding real degradation.
• Maintenance windows. Schedule windows for patching, backups, and batch processing, and make window creation part of the change process rather than a step someone remembers to do.
• One notification layer. Route alerts from every source into a single layer that groups related events, rather than letting three systems each own a phone number.

Measure alert volume per shift

Tuning needs a number the team can watch. Count alerts delivered to humans per shift, broken out by recipient and by rule, track how many led to any action, and track how the last several real incidents were first detected. If incidents are still found mainly by users calling in, the symptom checks are incomplete no matter how quiet the queue looks. Set a volume target the rotation considers readable overnight, agree on it with the staff who carry the phone, and treat sustained overage as a defect to fix, not a condition to endure.

Review the worst offenders monthly

Once a month, list the rules that generated the most notifications and decide on each one. The options are narrow on purpose: fix the underlying problem, lengthen or raise the threshold, downgrade it from page to ticket or to dashboard only, or retire the rule. Record the decision, the owner, and the date. That record is how a CIO shows leadership or an auditor that a quieter configuration was a reviewed engineering decision with named owners, not a shortcut taken to stop the phone ringing.

What to ask vendors and what to put in the contract

• Ask how the platform performs dependency suppression and deduplication, and ask to see it demonstrated against the agency's own topology rather than a prepared demo.
• Ask whether agency staff can create, tune, and retire rules themselves, or whether every change requires a vendor ticket.
• Ask whether alert or event volume affects licensing cost, since volume-based pricing creates pressure against sensible collection.
• For managed services, state in the contract who owns tuning, how often it happens, and what the agency receives in writing about alert volume and false positives.
• Require baseline tuning as an implementation deliverable rather than a later services engagement, and require export of alert and incident history in a usable format, including at contract exit.

HQT is a technology and software company focused on United States state and local government, with capabilities that include cloud and enterprise IT operations, monitoring design, and systems engineering.

Frequently asked questions