Bias testing for agency AI: what to measure and how often

Reference article · 5 min read

An AI system can report strong overall accuracy and still work poorly for part of the population it touches: right most of the time, but wrong in one direction for applicants who file on paper, or for people served in a language other than English. Bias testing makes that visible before it becomes a pattern of decisions the agency has to explain.

The work is ordinary measurement: define the decision and the harm, pick comparison groups you can lawfully form, measure error rates rather than overall accuracy, set the unacceptable gap before testing begins, and retest when the model or data changes.

Start with the decision, not the model

Write down what the system decides or recommends, and which error hurts more. A false positive on a fraud flag delays a payment to someone eligible; a false negative on a safety screening leaves a risk unaddressed.

Unit of analysis. Per application, per person, or per case. Rates shift with the choice, so fix it once.
The system as deployed. Measure the model output and the outcome after human review. If staff override more often for one group, the deployed process behaves differently from the model alone.

Choosing comparison groups from data you lawfully hold

Whether an agency may collect, hold, or use a demographic attribute depends on the program's authorizing law, state privacy law, and sometimes anti-discrimination law. Testing for disparity is often treated differently from using the attribute in a decision, but not automatically. Confirm with counsel and your privacy officer first.

Attributes already collected. Demographics reported to a state or federal partner may support testing with no new collection, subject to the purpose limits on that data.
Groups needing no sensitive attribute. Service area, application channel, language of service, and age band often surface the same access problems with fewer legal questions.
Proxy estimates. Inferring a characteristic from surname and geography is done in some regulatory settings, but the estimate carries its own error and does not belong in the operational record.
Small groups. Report counts beside rates, and say when a group is too small to support a conclusion.

Measure error rates, not overall accuracy

Overall accuracy is one number for the whole population, so it cannot show a gap. Split results by group and compute at least the following.

Selection rate. The share of each group receiving the outcome. A difference is a signal to investigate, not a finding: groups differ for reasons the program already understands.
False positive and false negative rates. Among those who should not have been flagged, how many were; among those who should have been, how many were missed. The harm usually lives here.
Positive predictive value. Among those flagged, the share flagged correctly. A queue right far less often for one group wastes that group's time.
Calibration. When a score says an outcome is likely, does it happen about that often in each group.

These measures generally cannot all be equalized at once when underlying rates differ between groups. Choose the one that matters most for this program, record why, and report the others anyway.

Results also depend on labels. Ground truth drawn from past decisions carries past practice forward, so audited samples and appeal outcomes are better sources than the system's own history.

Decide what gap is unacceptable before you test

Set the standard in the test plan, before results exist. Otherwise it moves to fit the number that arrives.

Thresholds with actions attached. What difference means monitor, what means investigate, and what means do not deploy or suspend use.
Relative and absolute. A large ratio on a small base can be noise, so require both a proportional and a minimum absolute difference.
Named decision makers. Program owner, CIO or AI lead, privacy, and counsel, with agreement on who can stop a deployment.
Remedies chosen in advance. Better training data, a different threshold, narrower scope, added human review, or declining to use the system. Adjusting outputs by group raises its own legal questions.

File the plan and the results with the impact assessment so they survive staff turnover.

Retest after changes, and on a set cadence

Model changes. Vendor version updates and retraining, which in hosted products can occur without a release the agency controls, so notice has to come from the contract.
Data changes. A new source, a redesigned intake form, a cleared backlog, or a change in how a field is coded.
Population and policy changes. New geography, new eligibility rules, a shift in volume or case mix.
Use changes. Staff leaning on the output more heavily than at launch changes the system even when nothing technical moved.

Between full tests, track selection and override rates by group so drift shows up as a trend. Log each test: date, model version, data window, groups, metrics, thresholds, result, and action.

What to ask vendors and put in the contract

Ask what bias testing exists, on what population, for which groups, using which metrics. A single accuracy figure is not an answer.
Ask whether the test population resembles yours. A benchmark run on a national sample may say little about your caseload.
Require access to inputs, scores, outputs, and model version identifiers in exportable form, so you can test independently.
Require notice before model version changes, a window to retest, and a remedy if results move materially.
Require cooperation with audits, documentation of the vendor's own testing, and retention of test records to match your schedule.

Vendor testing is evidence, not a substitute. The population is yours, and the explanation will be asked of you.

HQT provides technology and software services to United States state and local government agencies, including data analytics and advisory work on AI systems.

Frequently asked questions

How can we test an AI system for bias if our program does not collect race or ethnicity?

Many agencies can test using attributes they already hold, such as service area, application channel, language of service, or age band, which often reveal the same access problems. Programs that report demographic data to a state or federal partner may be able to use that data for testing, subject to the purpose limits attached to it. Proxy methods that infer a characteristic from surname and geography are used in some regulatory settings, but they produce estimates with their own error and should be labeled as estimates. Confirm with counsel and your privacy officer what your program may collect and use before the first test.

What is the difference between accuracy and bias testing?

Accuracy is a single number describing how often the system is right across the whole population, so by construction it cannot show whether results are uneven. Bias testing splits results by group and compares error measures such as false positive rate, false negative rate, positive predictive value, and calibration. A system can look accurate overall while producing most of its errors in one group.

How often should an agency retest an AI system for bias?

Set a regular cadence and a list of change triggers, then act on whichever comes first. Retest after a vendor model version update or retraining, a new or changed data source, a new population or geography, and a change in eligibility rules or how staff use the output. Between full tests, track selection rates and override rates by group so drift shows up as a trend rather than a surprise.

What size gap counts as too large?

No single number applies across programs, which is why agencies set the standard in the test plan before any results exist. Tie thresholds to actions: a range that means monitor, a range that means investigate, and a level that means do not deploy or suspend use. Express the threshold in both relative and absolute terms, since a large ratio on a small base is often noise, and name in advance who has authority to stop the deployment.

Can we require a vendor to prove its AI product is not biased?

You can require the vendor to document what testing it has done, on what population, for which groups, and on which metrics, and to deliver that documentation on request. You should not treat it as proof for your program, because the vendor's test population may not resemble your caseload. Build in the agency's own right to test: access to inputs, scores, outputs, and model version identifiers in exportable form, notice before model changes, and a remedy if results move materially.