Procurement language for AI features in software contracts

Reference article · 5 min read

AI features now arrive inside software an agency was already planning to buy. A case management system adds a summarization panel; a permitting platform adds a document classifier. The procurement question is usually not whether to buy AI, it is what the contract says about the AI that came bundled with everything else.

Standard software terms cover licensing, support, security, and data handling. AI adds risks those terms miss: training on agency content, silent model changes, outputs that are wrong in ways staff cannot easily spot, and data that is hard to take elsewhere. The clause topics below close those gaps. This is general guidance, not legal advice: your state's procurement code, records law, and any statewide AI policy shape what you can require, so confirm specifics with counsel and your procurement office.

Define the capability before writing terms about it

Vague definitions produce vague obligations. Have the vendor describe, in writing:

What the feature does. State it as an action on an input: classify an intake form, extract fields from a PDF, draft a response.
Where the model runs, who provides it, and what data leaves your tenant. Prompts, attachments, retrieved records, and logs each travel differently, and upstream provider terms flow down to you.
Whether a person reviews output before it has effect. That answer determines how much of the rest matters.

Attach that description as an exhibit so later obligations point at something concrete.

Data rights, output ownership, and no-training commitments

Three separate things need addressing, and vendors often blur them together.

Input ownership. Agency data stays agency data, including prompts, uploads, and retrieved records. Say plainly that submitting content to an AI feature does not change ownership.
Output ownership. Seek ownership, or at minimum a perpetual, irrevocable, royalty-free license to use outputs for any purpose. Copyright in purely machine-generated material is unsettled in the United States, so a broad license grant is more durable than assumed exclusivity.
No training or improvement use. Cover training, fine-tuning, evaluation, and the broad phrase service improvement, with flow-down to subprocessors. The usual counter is a carve-out for aggregated or de-identified data. If you accept one, define de-identification and exclude anything drawn from confidential or protected records.

Model change notice, version pinning, and accuracy commitments

Change notice. Require advance written notice before a material change to the model, the provider, or the prompting approach. Define materiality to include version changes, and negotiate the notice period rather than accept a generic right to modify the service.
Version pinning. Some vendors can hold a model version for a defined window; others cannot, because their upstream provider retires versions on its own schedule. Ask directly, and if pinning is unavailable, trade it for longer notice and a regression testing window.
Accuracy commitments with a measurement method. A promised accuracy figure means nothing without a defined test set, a definition of a correct result, a named scorer, and a cadence. Better: the vendor runs a documented evaluation on a sample of your data at stated intervals and delivers results to you.
Remedies. Tie sustained quality failures to a cure period, service credits, and a termination right. Credits alone rarely fix a program problem.

Human override, logging, and audit access

Human review. Name the decisions the system may not make alone, and require the interface to support reject, edit, and escalate, not only accept.
Disclosure. Require the product to mark machine-generated content for staff, and where appropriate for the public.
Logging. Require retention of input, output, model version, timestamp, and acting user, exportable in a usable format. One log serves records requests, audits, and incident review. Align retention with your records schedule.
Audit access. Full model inspection is rarely available; documentation rights usually are. Ask for system documentation, evaluation results, and any third-party assessments.

Exit terms for prompts, embeddings, and tuned models

Standard exit clauses cover the primary database and miss what the AI layer created. Add:

Export of prompt and output history with metadata, in a documented format, at no added charge.
Disposition of embeddings and vector indexes built from your data: export where feasible, certified deletion in all cases, including backups, on a stated schedule.
Ownership or destruction of any model tuned on your records, and of the tuning dataset. Decide which you want before signing, not during transition.
Rights to configuration you paid to develop: prompt templates, taxonomies, rules, and review workflows.

What vendors will actually negotiate

Usually movable: no-training language, an output license, notice periods, logging and export, deletion certification, disclosure, and human-in-the-loop requirements. Movable with effort: version pinning, evaluation cadence and reporting, remedies tied to quality. Usually not movable: uncapped indemnity for AI outputs, guaranteed accuracy numbers, and direct privity with the upstream model provider.

When the capability is bundled into a larger platform, the vendor will resist reopening the master agreement. An addendum covering only the AI features is easier to get signed. If the feature is on by default, require an administrative control to turn it off and a commitment that doing so does not degrade contracted core functions. At renewal, treat a newly added AI feature as a change that triggers the addendum, not as a free upgrade.

How to start

Inventory where AI already appears in systems you run, including features added by update.
Work with counsel on one reusable AI addendum rather than negotiating each deal from scratch.
Put the capability definition and evaluation method in the solicitation so responses are comparable.
Request standard AI terms and the subprocessor list during evaluation, and name an owner to read the change notices and evaluation reports after award.

HQT builds and integrates software for state and local government agencies, including work with agency teams on specifying and evaluating AI capabilities in systems they buy.

Frequently asked questions

What contract clauses should we add when buying software with AI features?

The core set covers ownership of inputs and outputs, a commitment that the vendor will not train models on agency data, advance notice of model changes, accuracy or quality commitments with a defined measurement method, human review and override requirements, logging and audit access, and exit rights for prompts, embeddings, and any tuned model. Most agencies package these as a single AI addendum that attaches to their standard terms. Start by making the vendor define the AI capability in writing, then attach that definition as an exhibit so each obligation points to something specific.

Can we require a vendor not to train its models on our data?

Usually yes. No-training language is one of the more commonly accepted AI terms, and it should cover training, fine-tuning, evaluation, and general service improvement, with the same obligation flowing down to any subprocessor or upstream model provider. Vendors often ask for a carve-out for aggregated or de-identified data; if you accept one, define de-identification in the contract and exclude anything derived from confidential or protected records.

What happens if the vendor changes the underlying AI model after we sign?

Without a clause, most vendor agreements let them change the service at will, including the model behind a feature. Require advance written notice for material changes, define materiality to include model version and model provider changes, and ask whether the vendor can pin a specific version for a defined period. If pinning is not possible because the upstream provider retires versions, negotiate a longer notice period and a window to run regression tests before the change reaches production.

The AI feature is bundled into a platform we already use. What can we do?

Vendors rarely reopen a master agreement for a bundled feature, but they will often sign a short addendum that applies only to the AI functions. Ask for that, plus an administrative setting to disable the feature and a written commitment that disabling it does not degrade the core functions you contracted for. Treat any newly added AI capability at renewal as a change that triggers the addendum rather than a free upgrade.

What should we require on exit when AI features are involved?

Beyond the usual data export, ask for prompt and output history with metadata in a documented format, disposition of embeddings and vector indexes built from your data, and ownership or certified destruction of any model tuned on your records along with the tuning dataset. Include certified deletion covering backups on a stated schedule. Also claim rights to configuration you paid to develop, such as prompt templates, taxonomies, and review workflows.