Where to run a language model: a decision guide for agencies

Reference article · 8 min read

An agency picks a first real use for a language model: summarizing incident narratives, drafting responses to records requests, sorting service requests. The question soon stops being which model is best and becomes where the model runs. That one is easier to settle before a pilot than after, because it determines which agreements, reviews, and controls apply to everything built on top of it.

Three arrangements are common. Use a commercial AI service reached over the internet. Use a model service offered by the agency's cloud provider and invoked inside the agency's own account. Or run an open-weight model on compute the agency controls. Any of the three can be the right answer for a given use case. What decides it is the data the use touches, the approvals the agency needs, and what the agency can operate over time.

A commercial AI service

A vendor product reached over the internet, either an application staff sign into or an interface developers call. The vendor runs the infrastructure, the model, and the updates. The agency accepts terms and begins.

Where it fits. Work built on information the agency would release on request anyway: drafting public communications, summarizing published documents, writing code that holds no sensitive configuration. There is nothing to stand up, so a pilot can start quickly, and new capabilities generally appear in these products before they appear anywhere else, which makes this a reasonable place to learn what current models can and cannot do.

Where it is limited. Sensitive content raises questions the agency has to answer in writing: where data is processed and stored, how long prompts and outputs are retained, who may read them, whether they are used to improve the vendor's models, and which subprocessors are involved. Terms vary widely between vendors and between tiers of the same product. Some vendors offer government offerings that hold authorizations under federal or state cloud authorization programs, which can support sensitive use, but an authorization covers a specific service at a specific level rather than a vendor's whole catalog, and status changes over time. Verify scope and current status against the program's published listing and against the contract.

Cost and effort. Usage-priced, lowest to start, with no infrastructure to run. The effort moves to governance: an acceptable use policy, staff training, a written record of what was approved for what data, and some way to learn whether staff are already using tools nobody reviewed.

A cloud provider's model service in your own tenancy

A managed model service from a cloud provider, invoked inside the agency's own account, subscription, or project. The agency's identity, logging, network, and key management controls apply as they do to other services in that environment.

Where it fits. Sensitive work at an agency already operating in that provider's cloud. The request stays inside a boundary the agency has documented and assessed, so the compliance argument reuses work that already exists rather than starting over. Provider agreements, including addenda covering regulated categories of data, may extend to these services, and the controls are familiar to the team already running the environment.

Where it is limited. The constraint here is usually availability rather than capability. Government regions often carry fewer models and features than commercial regions, and new releases reach them later or not at all. Quota and capacity limits apply and can bind at production volume. The data still leaves agency equipment and is processed by a third party under contract, so legal review is shorter but not skipped, and coverage has to be confirmed for the specific service rather than assumed from the master agreement. Each provider's model interface is its own, so an application written directly against it takes work to move. If the agency is not in that provider's cloud already, this option means adopting the cloud first.

Cost and effort. Usage-priced, with architecture, identity, and logging work up front and the ordinary cost of operating in that cloud around it.

A self-hosted open model

Open-weight models the agency runs itself, on its own servers or on cloud compute it controls, with an inference stack the agency operates.

Where it fits. Data that cannot leave a particular facility or network, sites with limited or intermittent connectivity, steady repetitive volume where per-request pricing accumulates, and workflows that need a fixed model version that will not change underneath a validated process. Retention and logging are entirely the agency's to set, since nothing leaves. Quality on classification, extraction, and routine summarization is frequently sufficient for this kind of work.

Where it is limited. This is an operating commitment rather than a purchase. Someone has to size and acquire accelerators, patch the serving stack, watch for degradation, run evaluations, and decide when to move to a successor model. An agency that can neither staff that nor contract for it ends up with an unmaintained system holding sensitive data. Open models are generally weaker than the largest commercial models on long open-ended reasoning and on complicated multi-step instructions. Reserved hardware also costs the same whether it is busy or idle, so low or uneven volume wastes it.

Cost and effort. Capital or reserved capacity up front, low marginal cost per request, and a standing labor line that is easy to leave out of an estimate.

How to decide

Work through these in order. The early questions eliminate options, and only the last ones compare what is left.

1. Classify the data, including what staff will paste in. If any of it is confidential, exempt from disclosure, or governed by CJIS, HIPAA, FERPA, or a state privacy statute, a general commercial service is out unless an authorized offering and an executed agreement cover that specific category. Data that is merely non-public is the middle case, decided by agency policy rather than statute.
2. Name the approver and the evidence each one needs. If counsel, a security office, or a CJIS systems officer must sign, ask what each will require and how long assembling it takes. An option that reuses an existing authorization or addendum normally reaches a decision sooner than one requiring a new assessment, and that difference belongs in the project schedule.
3. Check availability against the specific need. Confirm the models and features you intend to use exist in the region, offering, and account type you would actually buy, not only in the vendor's commercial product. If they do not, the option is unavailable whatever its other merits.
4. Size the workload from a real sample and price it both ways. Estimate requests per month and tokens per request from actual documents rather than from a guess. Usage pricing favors low or uneven volume; owned or reserved capacity favors steady high volume and pays back only at utilization the agency can demonstrate.
5. Name the operator by name. Write down the person, team, or contract that will run the system through staff turnover and model changes. If you are considering self-hosting and no name fits, that option is out. Managed services shift that burden to the vendor and raise the opposite question: what the agency does when a model version it depends on is retired.
6. Decide what happens when the network does not. If the work must continue during an outage, or happens at a site with unreliable connectivity, that argues for local inference regardless of how the cost comparison came out.
7. Settle the records question before launch. Prompts and outputs may be records under the state's public records act and the agency's retention schedule, and may be discoverable in litigation. Set retention, access, and logging first.

Nothing requires one answer for the whole agency. Splitting the decision by data classification is common: a commercial service for public information, and a tenancy-hosted or self-hosted model for regulated work.

What this means for your solicitation

Do not specify a hosting model unless law or policy forces one. State the constraint and let responders propose: the data classifications involved, the boundary the data must stay in, the authorizations required, and the retention and training prohibitions. Then ask each responder where the model runs, in whose account, what the provider retains and for how long, and how a version change is handled.

Require a cost model at pilot and at production volume so usage pricing and owned capacity can be compared on the same basis. Ask for evaluation evidence on tasks resembling yours rather than benchmark scores. Require an exit path: prompts, outputs, evaluation sets, and any tuning artifacts exported in usable formats. Keep the option to change hosting later by requiring that the application reach the model through an interface the agency can repoint.

Where HQT fits

HQT provides independent help with this decision: classifying the data, testing a candidate use case at small scale before the hosting question is fixed, and writing hosting and data handling requirements into a solicitation. HQT does not resell the model services or hosting it recommends.

Frequently asked questions