AI & Automation
AI security: prompt injection and other risks agencies should plan for
The new attack surface that language model applications create, and the controls that address it.
Applications built on language models introduce risks that traditional security programs do not cover. An agency deploying an assistant, a chatbot, or an automated document processor should understand them before launch.
Prompt injection
A language model follows instructions in its input. If an attacker can place text where the model will read it, in a web page it retrieves, a document it summarizes, or an email it processes, that text can redirect the model: leak data, take unintended actions, or produce harmful output. This is the most important new risk, and no complete technical fix exists.
Controls: limit what the model can access and do, treat retrieved content as untrusted, require confirmation for actions, filter outputs, and test with adversarial inputs before deployment.
Data leakage
Models can reveal information from their context, including documents a user should not see. Permission-aware retrieval and strict source scoping prevent this. Logging of prompts and outputs lets the agency detect it.
Over-reliance and automation bias
Staff may accept plausible outputs without checking. Interfaces that show sources, confidence, and uncertainty, plus review workflows for consequential outputs, reduce this.
Supply chain
Models, libraries, and hosted services change. Pin versions, test changes, and know which vendors process your data.
Denial of service and cost
Model calls are expensive. Rate limits, budgets, and abuse detection apply.
Governance
An inventory of AI systems, risk classification of use cases, and a review process. The NIST AI Risk Management Framework and published guidance on securing language model applications provide structure.
Testing
Include AI-specific tests in acceptance: injection attempts, permission boundary tests, harmful output checks, and load tests. Repeat when models change.
What this means for procurement
Ask vendors how they handle prompt injection, what the model can access and do, how outputs are filtered, and what testing they performed. A vendor who has not heard the term is not ready.