Security
Prompt injection is the new SQL injection, and we test for it
Any system that lets a language model read untrusted text has a new class of vulnerability. Here is how we test before launch.
Twenty years ago, applications that built database queries from user input were routinely broken by SQL injection. Today, applications that feed untrusted text to a language model face prompt injection: text crafted to redirect the model's behavior. If an assistant reads web pages, documents, or emails, it can be attacked through them.
What an attack looks like
A document uploaded for summarization contains hidden text instructing the model to ignore its rules and disclose other content. A web page retrieved for an answer contains instructions to recommend a product. An email processed for triage instructs the system to forward data. The model, following instructions as designed, complies.
Why there is no simple fix
The model cannot fully distinguish instructions from data in its input. Filters help; none are complete. The durable defenses are architectural.
What we do before an AI system goes live
Limit capability. The model can only access sources and take actions the task requires. An assistant that answers questions has no ability to send email.
Treat retrieved content as untrusted. It is quoted, bounded, and never granted authority over the model's instructions.
Require confirmation for actions. Anything with side effects goes to a person.
Filter and log outputs. Sensitive patterns are caught before display; everything is logged.
Adversarial testing. We run a library of injection attempts through every input path and record what happens. New attempts are added as they appear.
Permission checks. Retrieval enforces document permissions independently of the model.
For agencies
Add prompt injection testing to acceptance for any AI-enabled system, including features inside products you already own. Ask vendors what their system can access and do, and what happens when a hostile document is uploaded. Expect a concrete answer.