Engineering
Observability: knowing your application is broken before residents tell you
Logs, metrics, and traces are not a luxury for public-facing systems. They are how you keep a promise.
A resident who cannot pay a bill online at 9 p.m. does not file a ticket. They give up, or they call in the morning, or they complain publicly. The agency learns about the outage late and secondhand. Observability is the practice of learning about it first.
The three signals
Logs. What happened, in detail, with context. Structured, centralized, searchable, retained according to policy.
Metrics. Numbers over time: requests per minute, error rate, response time, queue depth. Cheap to store and ideal for alerting.
Traces. The path of a single request through every component. Essential when a slow page could be caused by any of five systems.
What to alert on
Symptoms, not causes. Alert when residents cannot complete the task: error rate above a threshold, response time above a threshold, a synthetic transaction that fails. Alerting on every underlying cause produces noise that gets ignored.
Synthetic monitoring
A script that submits a test form every few minutes from outside the agency's network is the simplest way to know the public experience is working. It catches expired certificates, broken integrations, and DNS problems that internal monitoring misses.
Dashboards for humans
One screen per service showing the golden signals: traffic, errors, latency, saturation. If it takes more than a glance to know whether the service is healthy, the dashboard is wrong.
On-call
Someone has to receive the alert. For agencies without 24-hour staff, that may be the vendor under a support agreement, with clear escalation. Define it before the first outage.
Our default
Every application we deliver ships with structured logging, metrics, synthetic checks, and a dashboard, from the first deployment. It is part of the definition of done, not an add-on.