AI impact assessments: a practical process before a system goes live

Reference article · 5 min read

Several states now require a written assessment before an agency puts an automated decision system into service for a consequential purpose, and require it to be revisited later. The names differ: algorithmic impact assessment, AI impact assessment, automated decision system review. Scope and triggers differ too, and legislatures keep amending them, so confirm the current requirement against your state statute or executive order, your central IT policy, and your counsel.

What varies less is the shape of the document. The questions it asks are the ones a careful program manager would ask anyway: what is this system for, who does it affect, what data does it run on, what happens when it is wrong, who checks it, and how would anyone notice if it drifted. Below is the structure most assessments follow, who signs, and what the work involves.

When an assessment is required

Most policies key the requirement to consequence, not technology. A system is in scope when its output materially informs a decision about a person's access to benefits, employment, housing, education, health care, licensing, or liberty.

• Purchased software counts. A scoring or triage feature inside a product you bought is in scope on the same terms as a model built in house.
• Recommendation still counts. A system that ranks, flags, or prioritizes work for a human decider is usually covered: the flag shapes the decision.
• Exemptions are narrow. Spell check, translation aids, and spam filtering are commonly excluded, but read the list as carefully as the definition.
• Changes trigger review. A new population, purpose, data source, or model version usually calls for an update rather than a new assessment.

Purpose, alternatives, and affected populations

State the problem in program terms, not technical ones: the backlog grows, staff cannot review every claim. Then state what the system will do, and whether it decides, recommends, or only sorts work.

Alternatives is the section agencies skip and reviewers question. Record what else was considered: leaving the process alone, changing the rules, adding staff, simpler rules-based automation, or a different product. Say why this option was chosen and what benefit is expected, in terms you can measure later.

Describe who is subject to the system and which subgroups could experience different results. Use counts the program already reports, and avoid estimates you cannot support.

Data sources, quality, and accuracy

List each data source by system of record, what it was collected for, whether that purpose covers this use, and how long it is kept. Be candid about quality: missing fields, inconsistent coding, records that reflect past practice rather than present policy, and populations thinly represented.

Accuracy belongs here because it depends on that data. Record what was measured, on what data, and whether that data resembles your population. A vendor benchmark run on a national sample may not describe your caseload. Treat error consequences separately: a false positive that delays a benefit and a false negative that misses a safety issue call for different thresholds and safeguards. Where feasible and lawful, report performance by relevant subgroup.

Human review, notice, and appeal

Write down where a person enters the process, what that person sees, and whether the person may depart from the output. Review is meaningful only if the reviewer has the time, the information, and the authority to disagree, and if overrides are recorded and counted.

• Notice. How an affected person learns an automated system was involved, in the languages the program serves, and what they can be told about the basis of the result.
• Appeal. Which existing hearing or grievance path applies, and what evidence is preserved for it.
• Records. Which inputs, outputs, and model versions are kept so a decision can be reconstructed later.

Monitoring and the decision to proceed

Assessments that end at launch age badly. Commit to specific measures, a cadence, a named owner, and a reporting path. Define what pauses the system: accuracy below a stated floor, a subgroup gap beyond a stated level, override rates that suggest staff do not trust the output, or a pattern in complaints.

The document then reaches a decision: proceed, proceed with conditions, or do not proceed. Conditions should read as commitments with dates. Signatures usually include the program owner, the agency CIO or designated AI lead, the privacy officer, and counsel, with some states adding a central review office and an entry in a public inventory. Set a refresh date, commonly annual, and define what counts as a material change in the meantime.

What to ask vendors and require in the contract

The vendor supplies inputs. The agency owns the conclusions.

• Ask for documentation of intended use and known limitations, test results including subgroup results, a description of training data, and version history.
• Ask how model changes are communicated and whether you may test one before it takes effect.
• Require disclosure of AI features, including any added after award, plus advance notice of material model changes and cooperation with audits.
• Restrict use of agency data for training without written permission, and require export of inputs, outputs, and logs.

How much work this represents

The first assessment takes the longest, usually several weeks of part-time effort across program, IT, privacy, and legal, most of it spent gathering facts rather than writing. Later ones move faster because the template and reviewer expectations are settled. Scale depth to consequence: a short form for low-risk uses, a full assessment for anything touching eligibility or enforcement. Start by inventorying where AI already runs, including inside software you have bought, then assess the highest-consequence system first.

HQT works with state and local agencies on AI governance documentation, system evaluation, and post-deployment monitoring.

Frequently asked questions