Specialist technology and software for state and local government info@hqt.orgCommunitiesGlossarySearch

Security

Prompt injection is the new SQL injection, and we test for it

Any system that lets a language model read untrusted text has a new class of vulnerability. Here is how we test before launch.

Twenty years ago, applications that built database queries from user input were routinely broken by SQL injection. Today, applications that feed untrusted text to a language model face prompt injection: text crafted to redirect the model's behavior. If an assistant reads web pages, documents, or emails, it can be attacked through them.

What an attack looks like

A document uploaded for summarization contains hidden text instructing the model to ignore its rules and disclose other content. A web page retrieved for an answer contains instructions to recommend a product. An email processed for triage instructs the system to forward data. The model, following instructions as designed, complies.

Why there is no simple fix

The model cannot fully distinguish instructions from data in its input. Filters help; none are complete. The durable defenses are architectural.

What we do before an AI system goes live

Limit capability. The model can only access sources and take actions the task requires. An assistant that answers questions has no ability to send email.

Treat retrieved content as untrusted. It is quoted, bounded, and never granted authority over the model's instructions.

Require confirmation for actions. Anything with side effects goes to a person.

Filter and log outputs. Sensitive patterns are caught before display; everything is logged.

Adversarial testing. We run a library of injection attempts through every input path and record what happens. New attempts are added as they appear.

Permission checks. Retrieval enforces document permissions independently of the model.

For agencies

Add prompt injection testing to acceptance for any AI-enabled system, including features inside products you already own. Ask vendors what their system can access and do, and what happens when a hostile document is uploaded. Expect a concrete answer.

Get in touch

Have a project or a solicitation coming up?

Talk with an HQT engineer. We respond to agency, corporate, and partner inquiries promptly.