HackerHACKER PAPER · 2025 · arXiv preprint
Defeating Prompt Injections by Design
What question does this paper answer?
When an agent reads email, web pages or files, external text can redirect its next action. CaMeL separates planning from a trusted user request from parsing untrusted content. An interpreter then tracks each value’s provenance and permitted readers and checks security policies before tool calls. This places part of the protection in the execution system around the model, rather than only asking the model to recognize malicious instructions. AgentDojo experiments show the value of this design alongside costs in task completion, policy maintenance, user confirmation and side channels.
What it contributes
- 01
Separating the planner from the model reading data is insufficient by itself: the plan can stay unchanged while untrusted content alters parameters such as a recipient or filename. CaMeL therefore checks tool-argument data flows and provenance as well (§2, §5; PDF pp. 3–11).
- 02
The paper separately evaluates benign task completion, behavior under attack and policy triggering in AgentDojo’s workspace, banking, travel and Slack tasks. Improved security does not mean every model retains its utility. The paper and appendices report failures involving actions determined by data, poorly documented APIs and insufficient context (§6, Appendices B/G/H; PDF pp. 11–19, 33–41, 52–124).
- 03
Provenance labels and policies help constrain disallowed data transfers, but protection depends on policies, tool annotations and the runtime implementation. Formal verification of the interpreter and its security properties remains future work (§9–10; PDF pp. 23–25).
Check the original passages
Pages count from the first page of the PDF file. Page jumps depend on the source viewer; section names and short quotes provide another way to find the passage.
the P-LLM only interacts with the user queryNo, prompt injection attacks are not fully solved.Guide source: arXiv v2 (24 June 2025) ↗
Read this paper alongside other sources
These readings connect the paper with specific arguments from talks and other sources.
Hacker News comments 13
Verified mentions linking to this paper, newest first. This is not the complete comment tree of any one submission. Recommendations, references, and criticism are labeled separately; comments remain in their original language.
> That does seem a little like solving the problems in AI by using more of it
Yes, and IIRC Google used this as part of a technique against prompt injection already [0], back when models were way more susceptible to it.
[0] Cf. CaMeL: https://arxiv.org/abs/2503.18813
See also "CaMeLs Can Use Computers Too: System-level Security for Computer Use Agents", https://arxiv.org/abs/2601.09923v1
> One threat we’re particularly focused on is prompt injection, and we handle it in layers. The model is trained to recognize and resist it. The harness marks anything coming from an untrusted source. Deterministic code checks the result. And an ensemble of classifiers runs where the agent can't reach them.
That "deterministic code" bit makes me wonder if they've implemented ideas from the DeepMind CaMeL paper: https://arxiv.org/abs/2503.18813 - my notes on that paper here: https://simonwillison.net/2025/Apr/11/camel/
Work is still being done on how to bulletproof input “sanitization”. Research like [1] is what I love to discover, because it’s genuinely promising. If you can formally separate out the “decider” from the “parser” unit (in this case, by running two models), together with a small allowlisted set of tool calls, it might just be possible to get around the injection risks.
[1] Google DeepMind: Defeating Prompt Injections by Design. https://arxiv.org/abs/2503.18813
Read 9 more comments
Prompt injection is _probably_ solvable if something like [1] ever finds a mainstream implementation and adoption, but agents not being deterministic, as in “do not only what I’ve told you to do, but also how I meant it”, all while assuming perfect context retention, is a waaay bigger issue. If we ever were to have that, software development as a whole is solved outright, too.
[1] Google DeepMind: Defeating Prompt Injections by Design. https://arxiv.org/abs/2503.18813
The solution is to not load it into context at all. I’ve seen a proposal for something like this but I can’t find it (I think from Google?). The idea is (if I remember it correctly) to spawn another dedicated (and isolated) LLM that would be in charge of the specific response. The main LLM would ask it questions and the answers would be returned as variables that it may then pass around (but it can’t see the content of those variables).
Edit: found it. https://arxiv.org/abs/2503.18813
Then there’s another problem: how do you make sure the LLM doesn’t leak anything sensitive via its tools (not just the payload, but the commands themselves can encode information)? I think it’s less of a threat if you solve the first problem, but still… I didn’t see a practical solution for this yet.
Regardless, here’s the CaMeL paper. Defeating Prompt Injections by Design (2025): https://arxiv.org/abs/2503.18813
Here’s a paper offering a survey of different mitigation techniques, including CaMeL. Design Patterns for Securing LLM Agents against Prompt Injections (2025): https://arxiv.org/abs/2506.08837
And here’s a high-level overview of the state of prompt injection from 'simonw (who coined the term), which includes links to summaries of both papers above: https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/
AI engineering, not ML engineering, is one way of framing that.
I don't write papers (I don't have the patience for that), but my work does get cited in papers from time to time. One of my blog posts was the foundation of the work described in the CaMeL paper from DeepMind for example: https://arxiv.org/abs/2503.18813
This problem remains almost entirely unsolved. The closest we've got to what I consider a credible solution is the recent CaMeL paper from DeepMind: https://arxiv.org/abs/2503.18813 - I published some notes on that here: https://simonwillison.net/2025/Apr/11/camel/
[1] https://arxiv.org/pdf/2503.18813
No, but it plays well in quarterly earnings. So expect a good bit, I'd say.
> The flaws are baked into the training data.
Not in terms of attack area.
In terms of hallucinations... ish. It's more baked into the structure than the training data. There's increasing amounts of work to ensure well-curated training data. Still, the whole idea of probabilistic selection of tokens more or less guarantees it'll go wrong from time to time. (See e.g. https://arxiv.org/abs/2401.11817)
In terms of safety, there's some progress. CaMeL is rather interesting: https://arxiv.org/abs/2503.18813
> "Trust but verify" applies.
FWIW, that applies to humans too. Nobody is error-free. And so the question becomes mostly "does the value overshadow the cost of the error", as always.
What changed is the new CaMeL paper from DeepMind, which notably does not rely on AI models to detect attacks: https://arxiv.org/abs/2503.18813
I wrote my own notes on that paper here: https://simonwillison.net/2025/Apr/11/camel/
Direct submissions and discussions (5)
Defeating Prompt Injections by Design · 1 points · 2025-04-21 ↗Defeating Prompt Injections by Design · 2 points · 2025-04-19 ↗CaMeL: Defeating Prompt Injections by Design · 71 points · 2025-04-19 ↗Defeating Prompt Injections by Design · 2 points · 2025-04-11 ↗Defeating Prompt Injections by Design · 1 points · 2025-03-26 ↗