Guides & Checklists
Building an AI test set from your own records to measure accuracy
How to assemble an evaluation set from real agency cases, set the answer key, and hold vendors to a measured accuracy number.
A vendor demonstration shows the product working on examples the vendor chose. That tells you it can work. It does not tell you how it will perform on the records your staff handle every day, which are older, messier, and formatted in ways nobody planned for. The way to find out is to run it on your own material and compare the output to answers you already know are right.
That material is a test set: a fixed collection of real cases from your records, each paired with the answer an experienced person says is correct. Building one takes program staff and not just IT, and it has to happen before the demonstrations. It lets you compare products on identical footing, hold a vendor to a measured result instead of an impression, and check later whether the system still performs as it did when you signed.
Define one item and what counts as correct
Narrow the task until a single item is unambiguous. An item is one input plus the expected output: one scanned form and the date of service to be extracted from it. Write the scoring rule down before anyone labels anything.
- Match rules. Whether a correct answer must match exactly, whether formatting variants count, and whether partial credit exists for multi-field extraction.
- Abstention. How you score the system saying it does not know. An abstention routed to a person is usually worth more than a confident wrong answer.
- Error types. Over-redacting and failing to redact a name are both errors, and they are not equivalent. Score them separately.
Draw a sample that includes the hard and rare cases
Pull from the system of record, across a span of time long enough to include format changes, staff turnover, and older conventions. Stratify so that categories, offices, and document types appear in roughly their real proportions. Then add cases on purpose:
- Items that generated complaints, appeals, or corrections in the past.
- Rare categories that appear seldom but matter when they do, including the exceptions your policy manual singles out.
- Poor scans, handwriting, forms filled out incorrectly, and documents in languages other than English.
Report the routine block and the difficult block separately. Folded into one average, a headline number hides the performance you most need to see. On size, the test is whether a handful of items could swing the result. If so, it is too small.
Establish the answers with staff who do the work
The answer key decides whether any of this means anything, and it is the expensive part. Have two experienced staff label each item independently, without seeing each other's work, and route disagreements to a third person who decides.
Track those disagreements rather than discarding them. An item two seasoned staff cannot agree on is not a fair test of software, and a pattern of disagreement usually points at guidance that is ambiguous and worth fixing anyway. Record the reason for each answer, then freeze the key with a version and a date.
Keep the answer key out of the vendor's hands
Separate the inputs, which a vendor has to process, from the key, which the vendor should never see. Once a vendor holds both, a strong score may reflect tuning to your test rather than capability.
- Hold back a portion. Reserve part of the set, shared with no one outside the agency, for the rerun later.
- Protect the inputs. Test data is agency data. Run it through your privacy and data sharing review, de-identify where you can, and put retention, deletion, and no training on your material in writing.
- Treat the set as a record. It is made of agency records and sits under your retention schedule and public records obligations, so confirm handling with your records officer and counsel.
Set the threshold before you see results
Decide what passing means while you are still uncommitted. Set a target for the overall score, a separate floor for critical categories, and a limit on the error type that would do the most damage. Say what the number is compared against, ideally your current manual process, which is often the harder half to measure.
Tie the threshold to the workflow. An accuracy level that is fine when a person reviews every output is not fine when outputs go out unreviewed. Write down in advance what a pass, a conditional pass with remediation, and a fail each lead to.
Put it in the solicitation and the contract
Say in the solicitation that finalists will run an agency-supplied test set, and describe the format, the time allowed, who observes, and how results factor into scoring. Ask each vendor whether the configuration used in the test is the one you would run in production, whether the system can decline to answer, and what happens to accuracy when the underlying model is updated.
Useful contract terms include the right to rerun the test with fresh samples during the term at no added charge, advance notice of model or prompt changes, export of raw outputs so your staff can score independently, and a link between re-test results and renewal.
Rerun it on a schedule
Results drift as vendors update models and your own forms and programs change. Rerun the set after an announced model change, before each renewal, and at a regular interval in between. Refresh part of it with recent cases each time while keeping a frozen core so scores stay comparable. Keep the scoring spreadsheet with the set and log every result with its date. A decline you can document supports a remedy or a decision not to renew.
HQT provides technology advisory and systems engineering services to state and local agencies, including evaluation and test design.
Frequently asked questions
How many records should be in an AI test set?
There is no fixed number that fits every task. The practical check is whether a handful of items could swing the score; if a few cases move the result, the set is too small. Balance that against labeling effort, since every item needs a verified correct answer from staff, and build the set in batches so the work stays manageable.
Should we send our test data to the AI vendor?
The vendor needs the inputs in order to run the test, but should never receive the answer key, because a vendor holding both can tune to your test. Treat the inputs as agency data: run them through privacy and data sharing review, de-identify what you can, and put retention, deletion, and no-training terms in writing. Hold back part of the set entirely so you have untouched material for a later rerun.
Who should decide the correct answer for each test item?
Staff who do the work, not IT and not the vendor. Have two experienced people label each item independently, then send disagreements to a third person to resolve. Record the reasoning behind each answer, and treat items staff cannot agree on as a sign that your own guidance needs clarification rather than as a fair test of the software.
What accuracy level should we require from an AI vendor?
There is no universal number, and naming one without context produces a score that means little. Set the threshold before the demonstration, compare it against how your current manual process performs, and set separate floors for high-consequence categories and for the error type that would do the most damage. Decide at the same time how much human review sits behind the output, because that changes what an acceptable score is.
How often should we rerun the test set after the system is live?
Rerun it after any announced model or configuration change, before each renewal decision, and at a regular interval in between. Keep a frozen core of items so scores stay comparable over time, and refresh part of the set with recent cases so it reflects current work. Log each result with its date, so a decline in performance is documented and can support a remedy or a decision not to renew.