Contracts & Procurement
Evaluating AI products: a guide for procurement and program staff
Questions to ask, evidence to require, and terms to include when an agency buys software with artificial intelligence in it.
Artificial intelligence now appears in nearly every software proposal, sometimes as the product and sometimes as a feature. Evaluating it requires questions that traditional software evaluations did not ask. This guide is written for the staff who run those evaluations.
Questions to ask
What does it actually do? A specific description of the task the model performs, its inputs, and its outputs. "AI-powered" is not an answer.
How was it tested? Accuracy measured on data like yours, with the methodology. Ask to run it on a sample of your own records.
What happens when it is wrong? Failure modes, how errors surface, and how a person intervenes.
Where does the data go? Whether your data is used to train the vendor's models, where processing occurs, and what is retained.
Can you explain a result? For any output that affects a person, whether the system can show why.
What is the human role? Which decisions remain with staff and how the interface supports review.
How is it monitored? What is measured in production and how the agency will know if performance degrades.
Which model, and who controls changes? Whether the underlying model can change without notice and what testing accompanies a change.
Evidence to require
Test results on your data, documentation of the model and its limits, a description of the data pipeline, security assessment or authorization for the hosting environment, and an accessibility conformance report for the interface.
Terms to include
- No use of agency data to train vendor models without written consent
- Notice and testing before material changes to the model
- Performance commitments with measurement methods
- Audit and explanation support for decisions affecting individuals
- Data location, retention, and deletion
- The agency's right to disclose the system's use publicly
Run a pilot
Score a pilot against the evidence above before committing. Real records, real staff, defined success criteria, and a fixed period.
Independent help
HQT provides independent technical evaluation of AI products for agencies and does not resell the products it evaluates.