Is that change real? Small counts, noise, and year-over-year swings

Reference article · 6 min read

A monthly report shows a measure moving from 10 to 13. Someone converts that to a 30 percent increase, and the number starts traveling: into a briefing memo, a press question, a council agenda item. The increase may be real. It may also be exactly what a small count does when nothing underneath it has changed. Telling those apart is ordinary analytic hygiene, and it belongs before the number leaves your office, not after an elected official has repeated it in public.

The checks below run in a spreadsheet, on any low-count measure: incidents, complaints, crashes at one intersection, grievances, failed inspections, denied permits. The last two sections cover how to state what you found and what to require of the systems that produce these numbers.

Convert the percentage back to counts

A percentage with a hidden denominator is a headline, not a finding. Write the raw numbers next to the percent: from 10 to 13, from 4 to 6, from 112 to 119. The conversation often changes on its own once people see that a 50 percent jump was two events becoming three.

• Do not lead with percent change when the base is small. State the counts, then add the percentage only if it clarifies something.
• Check what one event would do. If adding or removing a single record moves the percentage by ten points or more, that percentage is not a stable description of anything.
• Track the denominator separately. A rate can fall because the numerator fell or because population, calls for service, caseload, or inspection volume rose. Those are different stories leading to different decisions.

Expect small counts to bounce

Counts of events that occur more or less independently at a steady underlying rate do not repeat the same value month after month. They scatter around the average, and in relative terms the scatter is much larger when the average is small. A useful approximation: typical variation around the average is close to the square root of that average. An average near nine will routinely produce months around six and months around twelve. An average near one hundred will not routinely produce months of seventy or one hundred thirty.

That single approximation settles many arguments. Take the baseline average, take its square root, and see whether the new value sits within roughly two of those units. If it does, you have a reason to keep watching, not a finding to report.

Lengthen the window

One month against the prior month is the noisiest comparison available, and one year against a single prior year is not much better. More stable views:

• Rolling twelve months. Compare the most recent twelve months to the twelve before. Seasonality largely cancels, and no single month dominates.
• The same month across several years. Three or four prior Februaries tell you whether this February is unusual. One prior February tells you almost nothing.
• Year to date, with the cutoff stated. Reasonable for in-year reporting, as long as you say how many months it covers.
• Pooled categories. If counts per beat, per facility, or per category are tiny, report the total and treat the breakdown as descriptive only.

Rule out the mechanical explanations

Before attributing a move to program performance, check the causes that have nothing to do with performance.

• Calendar effects. Month lengths differ, as do the number of weekends, holidays, and business days.
• Reporting lag and backlog. Recent periods are often undercounted, and clearing a backlog inflates whichever month the clearing happened in.
• Definition or system changes. A new category, a revised intake form, a data migration, or a changed closure code moves a count without moving reality.
• Staffing and effort. More inspectors or a new enforcement push raises detected events. Detection is not incidence.
• One large event. A single incident with many victims, or one property generating many complaints, belongs in the narrative rather than buried in a total.

Draw simple control limits

A control chart is a line chart with three added lines, and any spreadsheet will build one. Choose a stable baseline of at least twenty-four periods. Draw the baseline average as a centerline. For counts, draw upper and lower limits at the average plus and minus three times the square root of the average, with the lower limit floored at zero. Then plot every period, including the new one.

Read it with a short set of signals: a point outside the limits, seven or eight consecutive points on the same side of the centerline, or a sustained run in one direction. Anything else is the measure behaving as it normally does. This method is deliberately slow to flag small real changes, which is the right tradeoff for a number that will be quoted publicly.

Word the finding honestly

Put the uncertainty in the sentence, not in a footnote.

Reported incidents rose from 10 in the prior month to 13. At this volume, month-to-month movement of this size occurs regularly without any underlying change. The rolling twelve-month total is essentially flat. We will continue monitoring and report if the pattern persists.

Habits worth adopting:

• Say what would change your conclusion, for example three consecutive months above the centerline.
• Prefer "consistent with normal variation" to "no change," and "we cannot yet distinguish this from normal variation" to "not statistically significant," which invites an argument about statistics instead of about the program.
• Apply the same standard to flattering numbers. A decline you never question is a credibility problem the next time you question an increase.

What to require of reporting systems

Ask during evaluation, and write into the contract:

• Dashboards show raw counts alongside every percentage change, and disclose the denominator behind every rate.
• Written definitions for each measure, with version history and advance notice before any definition, category, or schema change takes effect.
• Agency access to record-level data in an exportable format, so staff can recompute any published figure independently.
• Documented handling of late-arriving records, restated prior periods, and any small-cell suppression applied for privacy.
• No comparison view that presents percent change by itself.

HQT works with state and local agencies on data and reporting systems, including the definition and data-quality work that makes published measures reproducible.

Frequently asked questions