AI
Your vendor's new AI feature is a new system, not an update
Vendors are adding AI to platforms agencies already license. Here is what to test first, and what to leave switched off.
Software your agency already licenses is gaining AI features on the vendor's schedule. Permitting platforms, records systems, ERP suites, and productivity suites ship assistants and summarizers in ordinary releases. There is no solicitation and often no decision. The feature appears in a release note, or in an admin console toggle that is already on.
Treat a new AI feature as a new system, not an update. It brings a new data flow, a new failure mode, and sometimes a new subprocessor.
Find out what is already enabled
Start with an inventory rather than a policy. For each major platform, answer three questions: which features send content to a model, are they on now, and who can turn them on. The answer often differs by license tier and tenant setting.
Then ask the vendor, in writing:
- Which features are AI-enabled, and which are on by default in our tenant
- Where the model runs, whether prompts and documents leave our tenant or region, and what is retained
- Whether our content, including prompts and outputs, is used to train or improve any model
- Which agreement governs the feature, and whether a new subprocessor was added
- How we disable it, at what granularity, and whether disabling survives updates
The contract question matters most. Your master agreement and data protection terms probably predate the feature, and it sometimes arrives under a click-through a department administrator accepted for the agency. If your procurement relies on FedRAMP or a state program modeled on it, check whether the feature sits inside the authorized service boundary. Those authorizations cover specific services as assessed, not every capability added later.
Test the permission boundary first
The failure to look for first is not a wrong answer. It is a correct answer shown to someone who should not have seen it.
Assistants that search across a repository often build their index with a service account. If the retrieval layer does not filter results by the requesting user's permissions, the feature becomes a fast way to surface personnel files, draft legal memos, or closed session material to anyone with a login. The test is small: two accounts with different access, something distinctive in the restricted location, and a question from the account that should not reach it. We run it against our own reference builds first. Repeat it for anything that writes, so a summarizer cannot change a record the user could not edit by hand.
Build an evaluation set from real work
Benchmark numbers describe someone else's documents. Yours behave differently, usually in the awkward cases.
Ask the program staff who do the work to assemble real items with correct answers written down: permit records, code sections, service requests. Include the hard ones. Superseded policies still sitting in the repository. Two residents with similar names. Documents that contradict each other. Questions where the correct response is that the record does not say.
Score accuracy, and score failure behavior separately. When the system does not know, does it decline or produce something fluent and wrong. For public-facing work the second matters more, because a confident wrong answer costs staff time to catch and is harder to correct after someone relies on it.
Keep the set and rerun it when anything changes underneath. We keep ours in version control next to the prompts. Vendors update models without changing the interface, and a rerun is the only way to notice a regression.
What to leave off until tested
- Anything that writes to the system of record without a person approving the change
- Anything that sends content somewhere new, including automatic email summaries and meeting transcription
- Anything that drafts text sent to residents without review
- Transcription where closed session, privileged, or investigative material may be discussed
- Features that change search ranking or the order of a work queue without saying so
These are defaults until a test result changes them.
Turn it on narrowly and watch it
When a feature passes, enable it for one workgroup, for a named task, with a review step. Tell staff it is new, and make reporting a bad output a single click rather than a ticket. Log usage, and read the log after a few weeks, not at fiscal year end.
Check the interface against your accessibility obligations. Title II of the ADA covers state and local government services, and a Department of Justice rule sets WCAG 2.1 Level AA as the technical standard for covered web content and mobile apps, with compliance dates tied to entity size. Federal agencies have parallel obligations under Section 508. Chat panels and generated summaries are often the least tested parts of a product.
Then settle records handling. What these features create or summarize is generally subject to the same public records and retention rules as anything else staff produce, and prompts and outputs may themselves be records under your state law. Some states also require agencies to inventory or report their use of AI.
What this takes is modest: a current list of the AI features enabled in the platforms you license, a standard set of vendor questions, a default of off until tested, and an evaluation set owned by the people who do the work. A feature you did not procure still becomes yours to explain, to a resident, an auditor, or an elected official.