Automated decision systems: notice, explanation, and appeal rights

Reference article · 5 min read

When an agency uses software to decide who qualifies for a benefit, which properties get inspected first, or which cases are flagged for review, the software is performing part of a government decision. The duties attached to that decision do not weaken because a computer carried out a step. Notice, a stated basis for the outcome, and a route to challenge it follow the determination itself, not the tool used to reach it.

Specifics vary. Several states and localities have enacted statutes covering automated decision systems, and benefits, licensing, and enforcement programs carry their own notice and hearing rules. Confirm current requirements with counsel.

What counts as an automated decision system

Definitions differ, but converge on one idea: a computational process that produces a score, classification, recommendation, or determination that materially affects a person's rights, benefits, obligations, or treatment by the agency. Four points matter more than the phrasing.

• Rule-based systems count. A coded eligibility rule set in a legacy system makes determinations the same way a model does, and has for decades. Scoping a policy to "artificial intelligence" misses the systems that touch the most residents.
• Advisory output can still be a decision. If staff approve the recommendation in nearly every instance, the system is deciding in practice, whatever the label says.
• Purchased features count. Scoring and prioritization arrive inside case management, permitting, and fraud detection products, sometimes in a routine release. What the agency did not build, it still answers for.
• Scope turns on effect on people. Route optimization uses similar techniques without deciding anything about an individual. These duties attach where a person's benefit, license, penalty, or place in a queue is at stake.

Eligibility determination is the most common location, followed by enforcement case selection, risk scoring in child welfare, pretrial, and fraud review, and the prioritization behind waitlists and service queues. Most of it predates any AI initiative.

Notice that a system was used

Notice operates at two levels. At the program level, a published inventory tells the public which systems the agency uses, for what purpose, and whom to contact. At the individual level, the notice accompanying a determination should say that an automated process contributed to it, what that process did, what information about the person it used, and how to seek review.

Individual notice has to be usable: plain language, translation where language access rules apply, accessible formats consistent with Section 508 and WCAG expectations, and delivery through a channel the person uses. A disclosure buried in a portal's terms of use is not notice.

Explaining the basis in plain language

The explanation a resident needs is the reason for the outcome in their case, not a description of the algorithm. State the facts relied on, the source of each, the rule or threshold applied, and what would change the result. "The system determined you are ineligible" is not a basis. "Your reported household income, from the wage record dated as shown, exceeds the program limit for a household of your size" is.

Where a model produces a score, say what it measures, how it is used, and what it does not claim. Adverse action notices in most benefits programs already require a statement of reasons and the rule relied on. An automated step does not relax that, and neither does a vendor's confidentiality claim. Resolve that tension in the contract, not during a hearing.

Human reconsideration that is real

A route to a human means a named person with the authority and the information to reach a different result, not a queue that returns the same output.

• The reviewer can see and change the inputs. Most disputes are about a wrong or stale record, so a path to correct the underlying data must exist alongside the path to appeal.
• The reviewer is expected to disagree sometimes. Automation bias is well documented. Give reviewers time, training, and an explicit mandate, and watch override rates as a signal that review is working.
• Existing appeal machinery still applies. Fair hearings and administrative appeals do not change because a system was involved. The record should include the inputs used and the rules version in effect on the determination date.

Documenting the rules

Write the decision logic in plain language and keep it under version control, so the agency can reproduce what the system would have decided on any past date. Retain inputs, outputs, overrides, and the logic version as part of the case record for the program's retention period. Treat a change to decision logic as a policy change with a named approver. Test for disparate impact before deployment and periodically afterward, and keep the results.

What to ask vendors and put in the contract

Ask where the product scores, ranks, flags, or decides, and whether each point is configurable. Ask whether it can emit a per-determination explanation in plain language for the agency's own notice. Ask which fields feed each decision, how a past determination is reproduced after an upgrade, what disparate impact testing exists, and whether overrides are logged.

In the contract, require disclosure of every automated decision function, including ones added later; access to logic, inputs, and outputs sufficient to explain and defend a determination, with confidentiality protections rather than a refusal; agency approval before material changes to decision logic; cooperation with appeals, audits, and records requests; and export of decision records.

Start with an inventory, classify each system by its effect on people, and read the current notice language against what the consequential ones do. If nobody can write the plain-language rules statement, that gap is the first finding.

HQT builds and integrates systems for state and local agencies, including the logging and explanation features that make determinations reviewable.

Frequently asked questions