Vendor AI disclosure: what to require about AI inside products you already own

Reference article · 5 min read

Most AI governance work at state and local agencies is organized around a purchase: a new tool, a solicitation, a review board, a signature. The harder problem is the AI nobody bought. A permitting platform adds document summarization. An HR system adds resume ranking or interview scoring. Computer-aided dispatch adds call transcription and draft incident narratives. Utility billing adds anomaly detection on consumption data. An office suite adds an assistant that can read every file the signed-in user can read. None of that goes through procurement. It arrives in a release note, a feature flag, or a setting that defaults to on.

Vendor AI disclosure is the control that closes this gap. It is a standing obligation, not a one-time questionnaire: the vendor tells you in writing what AI functionality exists in the product you already license, what it does with your data, and what has changed since the last time they told you. The questions below are written for the version of this problem you are most likely to face, where the feature showed up in an update.

Ask about features, not about "AI"

A vendor can answer "we do not sell an AI product" truthfully while shipping three models inside the release you installed last month. Definitions vary, and marketing language moves faster than contract language. Get past that by asking feature by feature and function by function.

• Enumerate, do not characterize. Ask for a list of every product capability that uses machine learning, a large language model, or generative output, including features that are off by default, in beta, or limited to certain user roles.
• Include the quiet ones. Search, ranking, routing, matching, fraud and anomaly scoring, autocomplete, translation, redaction, transcription, and summarization all belong on the list even when the vendor does not market them as AI.
• Ask what is planned. Request the roadmap items expected to reach your environment in the next release cycle, so notice is not the first you hear of it.

The disclosure questions that matter

• Models and providers. Which models are used for each feature, who operates them, and is the vendor calling a third-party API, hosting an open-weight model, or running something they trained themselves?
• Subprocessors. Which entities receive agency data because of the AI feature, and are they listed in the current subprocessor schedule? A new model provider is usually a new subprocessor.
• Training and improvement. Is agency data, including prompts, uploaded documents, and outputs, used to train, fine-tune, or evaluate any model? Ask for a flat prohibition rather than an opt-out you have to remember to exercise.
• Inference location and retention. Where does processing occur, in which regions, and how long are prompts, outputs, and logs retained by the vendor and by any model provider? Ask whether any of it leaves the environment covered by your existing security authorization.
• Regulated data. Whether the feature can touch criminal justice information, protected health information, student records, or tax data, and how the vendor's arrangements handle that. Requirements in these areas change, so confirm current obligations with your state CJIS systems agency, your privacy officer, or counsel rather than relying on a vendor summary.
• Notice before change. How much advance written notice you get before a model, provider, or feature default changes, and through what channel. Release notes read by nobody are not notice.
• Controls. Whether the feature can be disabled tenant-wide, restricted by role, and left off by default, and whether the vendor can re-enable it without your action.
• Outputs and records. Whether outputs are stored in the system of record, whether they are distinguishable from human-authored content, and how they behave under your retention schedule and public records obligations.
• Evidence. Any accuracy testing, bias evaluation, red-teaming, or accessibility conformance documentation for the specific feature, not for the company.

What to put in the contract

Disclosure survives staff turnover only if it is written down. At renewal or on the next amendment, ask your counsel to consider terms that cover: a definition of covered AI functionality broad enough to include features added later; an affirmative duty to disclose new AI functionality with a defined notice period before it reaches your tenant; features off by default until the agency enables them; a prohibition on training or model improvement using agency data, flowed down to subprocessors; subprocessor change notice with a right to object; retention and deletion commitments that name the model provider; and a remedy if a feature turns on without notice, such as the right to require it disabled.

Fold it into cycles you already run

• Renewals. Add an AI disclosure attestation to the renewal packet. No attestation, no signature routing.
• Security review. Add AI-specific items to your existing vendor questionnaire and reassessment cadence so the answer is refreshed on a schedule, not on rumor.
• Change management. Put vendor release notes on someone's desk. Assign each major system an owner responsible for reading them and filing anything AI-related.
• Incident and records processes. Make sure your records officer knows which systems can now generate text.

Get it into the inventory

Many state policies and the voluntary NIST AI Risk Management Framework assume you can name your AI uses. Requirements differ by state and change over time, so check your own statewide policy. For each update-delivered feature, record the host system, the feature, the business owner, data categories involved, whether it influences a decision about a person, the current on or off state, the date you were notified, and the review date. A feature that arrived in a patch belongs in the same inventory as a system you competed.

HQT works with public agencies on technology advisory and systems engineering, including vendor evaluation and governance processes of this kind.

Frequently asked questions