AI
Running language models inside the agency's cloud boundary
For sensitive data, the model should come to the data. Here is what that looks like in practice.
Agencies ask whether they can use language models on documents that contain personal, health, or criminal justice information. The answer depends on where the model runs and where the data goes. For sensitive content, the workable pattern is to run the model inside the agency's own cloud environment so that nothing leaves the boundary.
The options
Public API services. Convenient and capable, but data leaves the agency's environment. Appropriate for public content and for vendors whose terms and authorizations meet the agency's requirements.
Cloud provider model services within the agency's account. Major cloud providers offer hosted models that run within the agency's own cloud tenancy and region, subject to the provider's authorization status. Data stays within the agency's boundary and is governed by the same controls as the rest of the environment.
Self-hosted open models. Open-weight models run on the agency's own compute, in the cloud or on premises. Maximum control, more operational work, and capability that has closed much of the gap with the largest models for many agency tasks.
What we do
For sensitive workloads we default to the second or third option, inside the agency's environment, with logging, network isolation, and access control applied like any other service. We evaluate whether a smaller open model handles the task well enough before assuming the largest model is required; often it is, and it is cheaper and faster.
Practical points
- Government cloud regions and authorization status vary by model service; verify before designing.
- Prompts and outputs are records; decide retention and access up front.
- Cost is per token; measure a realistic workload before committing.
- Model versions change; pin them and test before upgrading.
For procurement
Ask where the model runs, whose account it runs in, what the provider retains, and whether the agency's data trains anything. Write the answers into the contract.