Prompt injection: when data becomes instructions

The interactive below puts you at the controls of a six-stage agent security checkpoint. Pick an attack vector, toggle defenses one at a time, and watch the pipeline report its verdict. Use the Step button to move through stages manually, or let it run and compare verdicts as you adjust controls.

New tab

Demo

Prompt injection: an agent pipeline security checkpoint

Watch untrusted content ride into an agent pipeline and compete with the real task inside the model. Toggle four attack vectors and six layered defenses to see which combinations hold, which degrade gracefully, and which let the hijack through.

Built with Canvas 2D and vanilla JavaScript. Deterministic rule model over four attack presets and six defense toggles. Render and interaction by Pro Trailblazer.

What you're seeing

The pipeline runs left to right through six stages: INGEST, ASSEMBLE, INTERPRET, TOOL GATE, OUTPUT CHECK, and VERDICT. On the left, three input sources feed the : a SYSTEM GOAL (the operator's standing instructions), a USER TASK (the immediate request), and a RETRIEVED PAGE or TOOL RESULT (external content the agent fetched to do its job). The injected payload, shown as a red shard, rides inside that third source. Trusted content arrives as white-bordered packets; the injection glows red.

At the center, the model assembles all three inputs into its and interprets them together. The HUD stat labeled ATTACK REACHED MODEL shows whether the injected directive cleared the model's interpretation step. After the model, a TOOL GATE shows which tools are exposed and whether any of the active defenses block the tool call before it fires. An OUTPUT CHECK scanner sits between the tool gate and the action terminal. The VERDICT panel on the far right shows the final status (SAFE, BLOCKED, FLAGGED, SUSPECT, REDUCED, or HIJACKED), the severity, and two residual lines that explain what remained true even after defenses fired.

The CONCEPTUAL MODEL caveat on the defense panel is there for a reason. This is a deterministic rule model designed to teach the shape of the attack surface, not a live LLM inference. The influence scores and factor multipliers are illustrative. Treat the outcomes as a map of how each defense covers (or fails to cover) each attack path, not as a precision estimate of real-world success rates.

A few things to try:

Enable only "Content delimiters," then switch to Indirect injection. Notice that delimiters barely reduce the influence (factor 0.95) because the attack arrives inside second-hop content the agent fetched on its own. The model sees it inside the data section, not flagged as untrusted.
Switch to Fake tool result and enable only "Allowlist tools." The tool gate blocks the send_email call because it is not on the task allowlist. The BLOCKED verdict appears even though the model was still influenced. Then remove the allowlist and add only "Confirm actions" to see the same block come from a different checkpoint.
Try Data exfil request with Full access and no defenses. Then add only "Least privilege" and watch the severity drop from HIGH to MODERATE. The tool still executes, but scoped credentials mean only project notes are exposed, not the full note store. Then add "Output check" and watch the exfil link get caught before delivery.
Toggle "Trust boundaries" off to hide the trust-tier overlay. This is closer to what a model sees: all three input sources are present in one assembled context with no structural separator it can rely on.
Enable all six defenses against each attack preset. The strongest stack never reaches HIJACKED, but the residual line always notes that the model was still influenced. No stack reads "secure," only "reduced" or "blocked."

How prompt injection actually works

A language model processes its entire context as one flat sequence of tokens. The system prompt, the user message, and the retrieved document all get the same treatment: the model predicts the next token based on everything that came before. There is no hardware separation between "instructions I should follow" and "data I should read." That boundary is a convention, and the model was trained to observe it, but training is not a hard wall.

Prompt injection exploits that flatness. An attacker places instruction-shaped text inside content the model was asked to process, such as a retrieved web page, a search result, or a response. If the injected text is plausible enough given the model's training distribution (which includes vast amounts of instruction-following text), the model may weight it alongside the legitimate system instructions when deciding what to do next.

The demo models this as an influence score. Each attack type has a base influence (how instruction-like the injected text appears) and each defense applies a multiplicative factor. "Trust labels" and "Content delimiters" reduce influence by making the injected text look more like data. If the reduced influence clears a threshold, the model acts on the hijack. If it does not, the intended task runs instead. For the indirect injection preset, delimiters barely reduce influence (factor 0.95) because the attack arrives inside a page the agent fetched autonomously, which sits in the data tier where delimiters are already expected.

Once the model decides to follow the injected directive, the checkpoints outside the model are what matter. The tool gate checks whether the tool the attack requires is in the agent's current scope. The allowlist check compares the required tool against the task-specific allowed set. The confirmation gate holds sensitive tools (send_email, post_web) for before firing. The output scanner checks the final response against known payload patterns. Least privilege reduces what the tool can do even if the call goes through.

Why the attack surface grows with autonomy

A prompt-only chat session has a small injection surface: the model can produce bad output, but it cannot send email or post to a public board without tools. The surface expands as you add : read tools bring untrusted content in, write tools let the model act on what it read, and network tools let it exfiltrate data or interact with third-party systems. Each tool added to the agent's scope is a potential blast radius expansion if an injection succeeds.

The autonomy preset in the demo (Read only / Limited tools / Full access) shows this directly. With Read only, the agent has one tool: read_page. Attacks that require send_email or post_web degrade to output-only vectors, which are less severe. With Full access, every attack that clears the model hits a tool that can fire. The demo marks those results HIJACKED at HIGH severity when no other defense is present.

The indirect injection variant is worth examining separately. In the demo, it arrives via a second-hop fetch: the agent reads a partner page that links to another page, and the injection is on that linked page. The agent fetched it autonomously as part of the task. From the model's perspective, the content came from a source the system prompt implicitly authorized by giving the agent a browse tool. This is why indirect injection is the hardest to detect: the retrieval itself was legitimate.

For a broader picture of how agents are structured and what controls apply to them, the model context protocol post covers how tool exposure and server scoping fit into a standardized agent architecture. And for how these risks show up in practical deployments, see the agentic workflow guide, which walks through concrete workflow design choices that limit tool scope from the start.

Frequently asked questions

What is the difference between direct and indirect prompt injection?

Direct injection puts the malicious text somewhere the user controls, like the chat input itself. Indirect injection puts it in external content the agent fetches during a task: a retrieved web page, a search result, or a tool response. Indirect injection is harder to defend because the content source is at least partly outside the operator's control.

Can you train a model to ignore prompt injection?

Training and fine-tuning can reduce susceptibility, and some models are hardened to be less likely to follow instruction-shaped text in the data tier. But training alone is not a complete defense. The attack surface is too broad and the injection patterns too varied for any trained heuristic to cover reliably. Defense-in-depth outside the model is what the current security posture relies on.

Do content delimiters actually help?

Yes, for some attack types and not at all for others. Delimiters help the model recognize when it is processing untrusted content, which reduces the probability it treats that content as an instruction. But for indirect injection arriving in second-hop fetches, the content lands inside the data section where delimiters are already present, so they provide almost no additional signal. The demo models this with a factor of 0.95 for the indirect attack versus 0.80 for simpler vectors.

What does the output check actually do in the demo?

It pattern-matches the model's proposed output or tool call against a set of known payload signatures before delivery. In the demo, some payloads are marked detectable (the access phrase reveal, the collect.example exfil link) and some are not (the indirect fallback note). The scanner catches detectable payloads and redacts them, resulting in a FLAGGED verdict instead of HIJACKED. It cannot catch patterns it does not know about, and it cannot undo a tool call that already fired.

Why does the residual line always show something even when the attack is blocked?

Because blocking the action is not the same as removing the cause. When the tool gate or output check blocks a hijack attempt, the model was still influenced by the injected directive. The next agent turn, a different task, or a change in tool scope could let the same influence vector succeed. The residual line is a reminder that defenses reduce impact but do not eliminate the underlying condition: untrusted content reached the model's context.

Key takeaways

Defenses live outside the model, not inside it. The model itself has no reliable way to distinguish a malicious instruction embedded in a retrieved page from a legitimate one in the system prompt. That means the meaningful checkpoints are the tool gate, the output scanner, and the autonomy scope, not the model's own judgment.
Layering defenses reduces impact but no single layer is complete. Content delimiters barely slow indirect injection because the attack arrives in second-hop content the model fetches autonomously. Allowlisting tools stops tool-seeking attacks but leaves output-only vectors unchecked. A strong stack cuts severity at every stage without eliminating the underlying risk.
Least privilege is the highest-leverage single control. An agent that can only read the page it was asked to read cannot send email or post to a public board, no matter what the injected text instructs. Reducing the tool surface shrinks the blast radius before any other defense fires.
Output checks can catch what the tool gate missed, but they cannot undo an action that already fired. If an injected directive triggers a tool call and the tool executes, scanning the output afterward is damage assessment, not prevention.
The indirect injection variant is the hardest to detect because the malicious content is one hop removed from anything the operator put in the context. The agent fetches a linked page autonomously and the injection arrives inside content that the delimiters treat as data tier because the fetch itself was task-sanctioned.