From AI pilot to production: ownership, runbooks, and funding

Reference article · 5 min read

A pilot proves a tool can work. Production means the agency depends on it. Those are different claims, and the distance between them is organizational more than technical. Pilots often stall at that transition for reasons that have little to do with model accuracy: no one can say who owns the system, what happens when an output is wrong, or which budget line pays for it after the grant closes.

The discipline is the same one an agency applies to a permitting system or a dispatch upgrade, adapted for a system whose behavior can change over time and whose errors often look plausible. Treat it as a gate. If these pieces are not in place, the honest options are to extend the pilot or to stop it, not to let a prototype quietly become load bearing.

Name a business owner, not just a technical one

An AI system needs a named person in the program that uses it, not only in IT. The business owner decides what the tool is allowed to do, approves changes to its scope, and answers to leadership when results are questioned.

• Business owner. A program manager or division director who owns the outcome and the decision to keep, change, or retire the tool.
• Technical owner. The staff responsible for integrations, access, logging, and vendor coordination.
• Records and legal contact. Someone who can answer questions about retention and disclosure before a request arrives.
• Backup for each. A system that only one informed person understands is a single point of failure.

Write the runbook before go-live

The most common gap between pilot and production is the absence of a path for bad output. During a pilot, whoever built it fixes problems informally. In production, a caseworker or clerk needs written instructions.

A workable runbook covers how a user flags a questionable output, who receives that report and how quickly they respond, how an incorrect record already sent to the public gets corrected, what staff do when the tool is unavailable, when the matter escalates to the business owner or counsel, and the documented limits of the tool, including tasks it should never be used for.

Monitor the output, not just the server

An uptime dashboard will not tell you the tool has started producing worse answers. Plan two layers of monitoring and name who reviews each.

• Operational. Availability, latency, error rates, and cost per period, including usage-based charges that can rise without warning.
• Quality. Human review of a sample of outputs against a fixed evaluation set kept from the pilot, plus the rate at which staff override or discard results.
• Change control. Vendor model updates, prompt changes, and data source changes all alter behavior. Require notice, and re-run the evaluation set after each one.
• Cadence. Put a revalidation schedule in writing, for example quarterly and after any material change, and record who signed off on each review.

Records, disclosure, and the staff who arrive later

Public records and retention obligations differ by state and by record type, and they change. Ask your records officer and counsel early rather than after a request arrives. Treat prompts, outputs, logs, and the documents the system produces as potentially subject to retention schedules and disclosure, then decide deliberately what the system stores and for how long.

Plan for turnover. Written procedures, a short training module for new staff, and documentation of known limitations keep institutional knowledge from leaving with the people who ran the pilot.

Fund it as a program, not a project

One-time grant or innovation funds are a reasonable way to test something. They are a poor way to sustain it. Before production, build a full annual cost picture: licensing or usage charges, integration and hosting, vendor support, internal staff time for review and monitoring, periodic revalidation, and rework as processes change.

Then identify the recurring source. If the only plan is another grant, say so in writing and name the date by which a sustaining source must be found. Align contract terms and renewal options with the agency's budget cycle so a funding lapse does not become an unplanned outage.

Exit criteria and an honest shutdown

Write exit criteria at the start of the pilot, in terms the business owner accepts: the accuracy or turnaround improvement required, the maximum acceptable review burden on staff, the cost ceiling, and the fairness checks the program must pass. Decide in advance what result ends the effort.

Shutting down is not a failure if the learning survives. Before turning the system off, capture the evaluation set and its results, the process map the team built while documenting the current workflow, the data quality problems the pilot exposed, and a short memo on what did not work and why. Those artifacts often improve the underlying process regardless of technology. Export and retain records per schedule, revoke access, and tell affected staff what changed.

What to ask and what to require

Use procurement to close these gaps, not assume them.

• Ask how the vendor notifies customers of model or prompt changes, and how much notice is given.
• Ask what the agency receives on exit: data, logs, and configurations, and in what format.
• Require documented performance measures, support response times, and named escalation contacts in the contract.
• Require that agency data not be used to train models beyond the agency's own use without written consent.
• Require disclosure of subcontractors and hosting locations, and confirm alignment with your security review process.

HQT works with state and local agencies on the engineering and operations side of this transition, including ownership models, runbooks, monitoring design, and lifecycle planning.

Frequently asked questions