Designing human review for AI outputs in agency workflows

Reference article · 5 min read

A human in the loop is the control most agencies name first when they add an AI-assisted step, and the one most likely to fail quietly. It works only if the reviewer can see enough to judge the output, has time to judge it, and faces no cost for disagreeing. Without those conditions, review happens on paper only: someone clicks approve, and the agency has a record of oversight without the oversight.

Five design choices decide whether it is real: what gets reviewed, what the reviewer sees, how much time the work is given, what is recorded when a reviewer changes something, and what is done with the pattern of changes.

Decide what gets reviewed and what gets sampled

Reviewing everything does not scale, and reviewing nothing is not a control. Sort by the consequence of an error, not queue size.

Review every item when the output affects a person's benefit, license, penalty, or place in a queue, when it leaves the agency as a public statement or records release, or when a rule requires human determination.
Sample high volume, low consequence, reversible work: internal summaries, draft routing, first-pass classification a later step will catch.
Sample deliberately. Mix random picks with low confidence items, new document types, and the weeks after any change. Write the rate down and give it an owner, since sampling drifts toward zero as backlogs grow.

Put the evidence in front of the reviewer

A reviewer who has to leave the screen to find the source will approve instead. Each queued item should carry what is needed to check it in place.

The source, at the right place. The passage or record the output draws on, shown beside it, with a way into the underlying document.
The input as received, including recognized text from a scanned document, so garbled input is visible rather than inferred.
Version information: which model, prompt, rule set, and sources produced the item, so later analysis can separate one version's errors from another's.

A confidence score is not the probability that an answer is correct unless someone has checked it against verified outcomes. Describe the bands in plain words, test them against actual corrections, and never let confidence alone be the reason an item skips review.

Set time and workload expectations you can defend

Review is work. If it is not in the staffing plan, it comes out of time spent on something else, and care is the first thing cut.

Measure before setting a target. Time real reviews on real items, including hard ones. A quota derived from how fast someone can click approve makes the control a formality.
Make disagreeing as easy as agreeing. No preselected approve, no bulk approve in the review-everything tier, and an edit path no costlier than acceptance.
Plan for fatigue. Cap session lengths, rotate assignments, and use spot checks of approved work to detect a queue being cleared rather than read.

Record the override, not just the outcome

When a reviewer edits or rejects an output, capture it as structured data, not as a changed document with no history. A usable record holds the item and workflow identifiers, the model, prompt, and rule versions in effect, the proposed and final outputs, the reviewer, the times received and decided, the decision type, and a reason code.

Keep the reason codes short and design them with the reviewers. Wrong source retrieved, output not supported by the source, missing context, format or tone, and underlying record is wrong are categories you can act on. A code that says only "incorrect" is not.

These logs are agency records. They may fall under retention schedules, may be disclosable under public records law, and can surface in appeals and audits. Settle retention and handling with your records officer and counsel before go-live.

Use disagreement rates to retune the system

The pattern of reviewer changes is the most useful measurement the workflow produces.

Break the rate down by document type, program, unit, reviewer, source, and time. One overall number hides the category where the system fails.
Treat a very low rate as a question. It may mean the output is good, or that review is not happening. Blind double review, where two people judge the same item independently, tells the two apart.
Watch for step changes after updates. Model updates, prompt edits, and new source content all move the rate, so require notice of vendor changes.
Send findings to whoever can fix the cause: retrieval problems to whoever maintains the index, wrong records to the data owner, format complaints to whoever writes the instructions. Decide in advance what a sustained rise triggers, up to suspending automation for that category.

What to ask vendors, and how to start

Ask to see the review interface during demonstrations, with messy examples rather than curated ones:

How does a reviewer reach the source evidence for an output, and in how many steps?
How is any confidence score computed, and has it been calibrated against verified results?
What is captured for each review, and can we export the full log, including overrides and reason codes, in an open format?

In the contract, address ownership of review and override data, export during the term and at exit, notice of material model changes with a right to test first, and the access needed to explain an output affecting a person.

To start, take one workflow with a clear owner. Review every output long enough to build a baseline by category, publish the rules for review and sampling, then move part of the queue to sampling, keeping the ability to reverse that.

HQT works with state and local agencies on AI-assisted workflows, including review interfaces, override logging, and measurement.

Frequently asked questions

What does human in the loop actually mean for an AI system in a government agency?

It means a person reviews the system's output and can change or reject it before it takes effect. The label alone proves nothing. The review counts only if the reviewer can see the source evidence, has time to check it, and can record a different result without penalty. Agencies usually document which outputs get individual review, which are sampled, and what reviewers are expected to verify.

Do we have to review every AI output, or can we sample?

That depends on the consequence of an error and on any rule or policy that requires a human determination. Outputs that affect a person's benefit, license, penalty, or place in a queue, and anything released publicly, are normally reviewed one by one. Lower-consequence, reversible work can be sampled, provided the sampling rate is written down, aimed at risky cases as well as random ones, and assigned to a named owner.

What should we log when a reviewer overrides an AI output?

Log enough to reconstruct the decision later: the item, the model, prompt, and rule versions in effect, the proposed output, the final output, the reviewer, the timestamps, the decision type, and a reason code from a short list. The reason codes are what make the log useful, because they group failures into causes someone can fix. Treat the logs as agency records and set retention with your records officer.

What is a good override rate for an AI-assisted workflow?

There is no published benchmark that transfers across agencies, document types, and use cases, so read the rate as a trend rather than a target. A rate that rises after a model change or new source content points to a regression worth investigating. A rate near zero can mean the output is reliable or that review is not really happening, and blind double review of a small sample helps tell those apart.

Are AI review and override logs subject to public records requests?

Often yes. Records an agency creates and keeps in the course of business are generally covered by state public records law, and review logs can also appear in appeals, audits, and litigation. Exemptions, retention schedules, and handling practices differ by state and by record type, so confirm the specifics with your records officer and counsel before the system goes live.