Before-and-after comparisons: what a pre/post number cannot tell you

Reference article · 5 min read

A program launches in July. In January, someone asks for proof that it worked. The evidence on hand is a chart with two bars: the twelve months before and the twelve months after. The second bar is lower. The question is whether that gap is the program.

A before-and-after gap is not worthless. It is the starting point for most real evaluations. But on its own it cannot separate the program's effect from everything else that changed at the same time, and pre/post numbers travel further than their evidence supports. They end up in budget testimony, grant reports, and council presentations, where correcting them later costs more than getting them right the first time. Most of the improvements below are cheap, and several can be done with data the agency already has.

What the gap actually contains

The difference between two periods is the sum of every cause operating between them. The program is one of those causes. The others are:

Secular trend. The measure was already moving before the program started. Caseloads, incident categories, application volumes, and call counts drift for reasons that have nothing to do with any local initiative.
Regression to the mean. Programs are usually launched after a bad stretch. Unusually bad periods tend to be followed by less unusual ones even with no intervention, so part of any improvement is the statistical echo of the spike that justified the program.
Selection into the program. If participants volunteered, were referred, or were screened in, they differ from non-participants in ways that also predict the outcome. Motivated participants improve more than average whether or not the program works.
Composition change. The population being counted shifted. A new eligibility rule, an annexation, an office closure, or a demographic change alters the denominator quietly.
Measurement change. The definition, the system, or the workflow that produces the number changed. This is the most common problem and the easiest to miss.

Counting changes are the usual culprit

When a new system launches, the count changes on day one for reasons unrelated to outcomes. Records get deduplicated during migration. Categories are consolidated or split. A field that was optional becomes required, so records that used to go unclassified now appear in a category that looks like it grew. Timestamps move from the moment staff entered the record to the moment the event occurred, which shifts events across period boundaries. A backlog cleared at cutover inflates the first months and deflates later ones.

Before comparing anything, reconstruct how the number was produced in each period. Ask what query defines it, which statuses are included, who could enter a record, and whether any of those answers changed. If the definition changed, the comparison is between two different measures, and no statistical method will repair that.

Cheap designs that beat pre/post

None of these require a randomized trial or an outside evaluator.

Comparison areas or units. Find districts, offices, counties, or facilities that did not get the program and track the same measure over the same period. If the untreated units moved the same way, the trend was not yours.
Interrupted time series. Instead of two averages, plot many periods before and after, monthly or weekly. Fit the pre-period trend, project it forward, and compare actuals to the projection. This separates a level shift at launch from a change in slope, and it exposes trends that two bars hide.
Difference in differences. Combine the two above: compare the change in treated units to the change in comparison units. It absorbs anything that affected both.
A falsification outcome. Pick a measure the program should not affect and check it over the same window. If it moved identically, suspect a common cause such as a reporting change.
Dose response. Where participation varied, check whether units with more exposure changed more. A flat relationship is a warning.
Pre-registration of the metric. Write down the measure, the query, and the comparison window before launch. This costs nothing and removes the strongest objection to any finding.

What to ask vendors and what to put in a contract

Measurement usually depends on the vendor, so handle it during procurement rather than at evaluation time.

Ask, in writing and at the field level, how the system defines each metric you intend to report.
Ask what changes during migration: deduplication rules, category mapping from the legacy system, and how historical records are backfilled.
Require export of row-level data, not only dashboards, with stable identifiers and event timestamps.
Require notice before any change to a metric definition, category list, or default report, plus a documented version history of those definitions.
Require that baseline data be extracted and archived before cutover, under the definitions in use at that time.
Do not accept vendor-produced before-and-after figures as performance evidence unless your own staff can reproduce them from raw data.

Reporting the result honestly

State the design, not only the number. A sentence such as "calls dropped after launch, and the same measure in three comparison counties did not" is more persuasive to a skeptical auditor than a percentage with no design behind it. Where you cannot rule out a competing explanation, name it and say what it would take to rule it out. If a grant or statute sets a specific evaluation standard for your program, confirm the current requirements with the funding agency or your counsel, because those rules change.

HQT works with state and local government agencies on data and analytics work, including metric definitions, baseline extracts, and evaluation designs meant to survive outside review.

Frequently asked questions

Is a before-and-after comparison ever good enough to show a program worked?

It can be adequate when the effect is large and immediate, the measure is defined identically in both periods, and no plausible competing cause changed at the same time. Those conditions are rare, and most program launches coincide with a new system, a new policy, or a period that was unusual enough to prompt the program in the first place. Treat a pre/post gap as a prompt to investigate rather than as a conclusion, and state the competing explanations you could not rule out.

What is an interrupted time series and can our staff do one?

An interrupted time series plots the same measure over many periods before and after the launch, then uses the pre-period trend to project what would have happened without the program. Instead of comparing two averages, you compare actual values to that projection, which shows whether the level shifted at launch, the slope changed, or neither. A simple version can be done in a spreadsheet with monthly values and a fitted trend line, and it is far more informative than two bars.

How do we find a comparison group if the program went agency-wide?

Look for variation in timing rather than in coverage: if the rollout happened in waves by office, district, or facility, the units that started later serve as comparisons for the earlier ones. Peer jurisdictions that publish the same measure are another option, as are neighboring counties or similarly sized cities. If no comparison exists at all, say so explicitly and rely on an interrupted time series plus a falsification outcome.

Our numbers changed right when the new system went live. How do we tell a real improvement from a counting change?

Reconstruct the definition used in each period, including the query, the included statuses, the category list, and which timestamp the record is dated by. Then check a measure the program should not have affected: if it shifted at the same moment by a similar amount, the change is most likely in the counting. Where possible, recompute the post-period number using the old definitions and the pre-period number using the new ones, and report both.

What should we require from a vendor so we can evaluate the system later?

Require written, field-level definitions of every metric you plan to report, row-level data export with stable identifiers and event timestamps, and advance notice of any change to metric definitions or category lists. Require that baseline data be extracted and archived before cutover under the definitions in use at that time. Also require enough documentation of migration rules, such as deduplication and category mapping, that your staff can reproduce any figure the vendor reports.