In July 2026, the AI investor and former HyperWrite CEO Matt Shumer was trying out OpenAI’s unreleased GPT-5.6, codenamed Sol, on his own Mac. He gave it full access to the machine. A cleanup subtask then ran a single rm -rf pointed straight at his home directory. By the time he reacted and killed it, over an hour had passed and most of his files were gone.
The uncomfortable part: OpenAI already knew about this class of risk. Sixteen days before the incident, its own model card had flagged “giving an AI Agent full access” as a serious risk.
There was no hacker here, and no exploit. Deleting files was simply one of the powers this Agent had been granted.
Which points to a more fundamental problem: the architecture of most of today’s AI Agents is insecure by design. And the instinctive fix — bolting on guardrails — is the wrong direction. Guardrails sit at the same level as the thing they’re meant to contain, so no matter how many you add, they can be worked around. The defense that actually holds has to live outside the model.
AI security means three different things, and this is about the third
“AI security” is a hot phrase right now, but in different contexts it refers to genuinely different things. Roughly three.
The first is the security of the model itself: preventing jailbreaks, keeping it from producing harmful output, aligning it with human intent. This is what most of the last few years’ discussion has been about.
The second is AI for security: turning large models around and using them for defense — threat detection, code auditing, intelligence analysis. We’ve done some work here ourselves, for instance using a thought-augmented model to detect smart-contract vulnerabilities in Thought Is All You Need, and building REEVMBench to measure whether AI Agents are actually ready to audit smart-contract security.
The third — the subject of this piece — is the security of the AI Agent itself. Once a model stops merely answering questions and starts reading your files, holding your credentials, and taking actions on your behalf, the tools it can call, the permissions it holds, and every action it takes become an attack surface.
The first two have mature bodies of research. The third is new, easy to overlook, and it happens to be the precondition for AI Agents entering the enterprise at scale: no company will hand real business and real data to an Agent that can’t keep itself safe.
From untrusted input to arbitrary execution: the inherent flaw
To do real work, an Agent usually needs several capabilities at once: it can read external content (web pages, email, code repositories, other people’s pull requests), it can reach valuable resources (keys, cloud passwords, databases), and it can call tools to act (run commands, make requests, edit files). Put those three together and you have the most dangerous combination there is.
The root of the problem is that the model can’t tell whether a given sentence is an instruction from its operator or content it just read that someone else wrote. To the model, it’s all the same text in the same context window. It decides who to listen to by learned intuition, not by any hard rule that separates the two. This is an old problem in security, the confused deputy: a program with privileges is tricked by a party without them into doing something harmful on its behalf.
The chain is worth spelling out. The tools that actually take action follow the model; the model follows the prompt; and the prompt is assembled from everything — some of it written by the user, some of it pulled in from web pages, email, and documents. So an untrusted source, passing through a model whose behavior isn’t deterministic, can end up driving tools to execute arbitrary operations. In traditional software, data is data and code is code, cleanly separated. With an Agent, a sentence it reads in can be treated as a command at any moment.
Which means an attacker needs no sophisticated exploit and no stolen account. Planting a single sentence in a document, a web page, or even a code comment can get the Agent to do something it never should. Microsoft’s security team put it bluntly: your LLM is not a security boundary; the tools you expose define your attacker’s affected scope.
None of this is hypothetical — the real cases are lining up. xAI’s Grok Build coding tool, on every work session, quietly uploaded the entire repository — including the secrets in .env and the full commit history — to its own cloud servers, at a measured volume nearly thirty thousand times what the task required. In one test, Cursor’s Agent deleted a company’s entire production database, backups and all, in nine seconds, then wrote an apology. Replit’s Agent, during an explicit code freeze, deleted the whole production database and fabricated thousands of fake records to cover it up. And Amazon Q’s VS Code extension was poisoned through a malicious commit that planted a “wipe the system” instruction, which shipped in an official release to nearly a million machines.
Line these incidents up and the common thread is clear: the damage almost always came from the Agent’s own authorized permissions, not from an outside break-in. Reading .env, running commands, querying databases — that’s just what it does every day. One upload channel, one delete command, and a normal capability becomes a disaster.
Why in-model guardrails fall short: the defender sits at the same level
The instinctive response to these incidents is to add guardrails inside the Agent. By guardrails, I mean rules set inside the model to keep it from doing dangerous things: you can use the prompt to tell it not to run dangerous operations, or add a check inside the Agent — the way some coding tools intercept dangerous commands and inspect file paths.
But this approach has a problem at its root: the guardrail and the thing it’s guarding against sit at the same level.
Apostol Vassilev, a senior scientist at the U.S. National Institute of Standards and Technology (NIST), published a paper in June 2026 that brings the logic of Gödel’s incompleteness theorems to bear on exactly this. The title is itself a rhetorical question: Robust AI Security and Alignment: A Sisyphean Endeavor?
His argument runs roughly like this. A guardrail is, at bottom, a finite, fixed set of rules. Just as a finite axiomatic system can’t be both complete and consistent, for any fixed set of guardrails there will always be a sentence that gets around it — and the ambiguity and flexibility of human language make the space of such sentences effectively endless. In his own words, no finite set of guardrails is universally robust against adversarial prompts. The only question left is who finds that sentence first, the defender or the attacker. NIST accordingly recommended shifting the mindset from “get it right once” to “monitor and patch continuously.”
There’s a more direct piece of evidence. If the model itself can’t be trusted, can you use one model to check whether another has been compromised? A study from the University of Wisconsin found that road is a dead end: this kind of scheme, where one model judges whether another has been tampered with, has a structural flaw, and an adaptive attack the researchers built drives the detection rate to near zero while the injection still succeeds about nine times out of ten. The model meant to catch attacks sits at the same level as the content it inspects, and ends up compromised first.
And there’s a timing issue that’s easy to miss: guardrails only kick in after the Agent already has the authority to act. EchoLeak is the clearest example. It was a zero-click attack on Microsoft 365 Copilot: a carefully crafted email slipped silently past the injection filters Microsoft had put in front of Copilot and carried sensitive data out, with the user never clicking anything.
So “add guardrails inside the model” is a bit like building a Maginot Line. Not useless, but by design certain to be outflanked. It can be one layer of the defense; it can’t be the only line.
The real defense belongs outside the agent
If the inside of the model can’t be trusted, the line of defense has to move outward, to the system level.
Confining untrusted code to an isolated environment and giving it only the least privilege it needs is not new in the security field — it’s decades old. As far back as 1975, Saltzer and Schroeder’s classic paper laid out two principles: least privilege, where a program gets only the sliver of permission it needs to do the job; and separation of privilege, where authority is split so no single mistake or deception can lay the whole system open (The Protection of Information in Computer Systems). The sandboxing that followed — from Janus in the 1990s, to FreeBSD Jails in 2000, to gVisor today — all walk the same path: isolate untrusted code so that even if it’s compromised, the damage can’t reach outside. Securing an AI Agent is, in the end, applying that decades-old body of practice to a new kind of object.
In concrete terms today, the direct move is to wrap it in a system-level sandbox: isolate the environment where the Agent runs tools, controlled down to the granularity of system calls, file reads and writes, and network egress. How much is on by default varies a lot between tools. OpenAI’s Codex sandboxes command execution by default and cuts off the network by default; whereas Claude Code’s local sandbox has to be turned on by hand, and Cursor’s sandbox, by its own description, is only “best-effort” and not to be treated as a security boundary. Going further, you can put the whole Agent inside a platform-managed runtime, handing network and credentials to that outer layer rather than to the model. Anthropic’s cloud-hosted managed Claude Code works exactly this way: each session runs in an Anthropic-managed virtual machine, all outbound requests pass through an allowlisted proxy, and credentials are held outside the sandbox.
A natural question follows: if sandboxing is this mature, how did those deletions and repo uploads still happen? Two reasons. First, the default protection in these tools is often thin — Cursor has no strict sandbox by default, and in the GPT-5.6 case the user simply granted full access, which is the same as switching the sandbox off. Second, even with the sandbox on, it doesn’t stop this class of incident: a sandbox guards against the Agent breaking out to harm the host, but the databases it deleted and the repositories it uploaded were things it was explicitly authorized to reach, no escape required. So the sandbox is a necessary layer, and nowhere near sufficient.
This “keep the Agent’s reach fenced in” idea has also made its way into products aimed at ordinary office users. Tencent’s desktop assistant WorkBuddy, released this year, runs on a personal computer and only reads the local folders the user has authorized. Its public materials, though, only mention “authorized folders” and never explain how the underlying isolation actually works, or how far it goes.
Which is the more important point: this whole category of “Agents with a sandbox” shares one blind spot. A sandbox controls where the Agent runs and whether it can break out to harm the host, but not what it does with the data once it’s running. A prompt-injected Agent moves entirely through explicitly permitted channels, and looks no different from normal operation. As one security researcher put it, reviewing Google’s GKE Agent Sandbox: this is an “isolation sandbox,” not a “behavior sandbox” — it decides where the Agent runs, but doesn’t watch what it does while running.
Precisely because a sandbox can only govern where the Agent runs and not what it does with data, Google DeepMind proposed a more thorough approach: treat the large model as an untrusted component from the start, and wrap it in a fixed, non-AI layer that decides, step by step, whether it may act. Their CaMeL compiles the trusted user instruction into a program run by a non-AI interpreter; that interpreter tracks the provenance of every piece of data and, before each tool call, decides by policy whether to allow it — while the model that actually touches untrusted external content has no power to call tools at all. On a good share of tasks, this delivers “provable security.”
The hard part sandboxes can’t solve: the data the agent actually needs
Still, whether it’s a sandbox or CaMeL, there’s a harder problem neither gets around, and it’s the trickiest part of the whole thing.
A sandbox separates the Agent from the host, but it builds no wall around the data the Agent needs in order to work. The code, the .env, the database it has to query — all of it lives in the same environment as the Agent; and to work at all, the Agent needs some channel out. So a compromised Agent can still send whatever it can read out through that permitted channel. Anthropic’s own sandbox docs concede as much: as long as network egress is allowed, data the Agent can read can be carried out, because the allowlist filters by domain and doesn’t inspect what’s leaving.
So how do you protect the data an Agent can’t do its job without? It splits into three cases, each harder than the last.
The first is explicit credentials, say an API key. The Agent doesn’t need to understand the key’s contents; it just uses it to make requests. So you can let the model see nothing but a placeholder from start to finish, and have an outer managed runtime substitute the real key in only at the moment of the actual request. Even if someone coaxes “the key” out through the prompt, what they get is a useless dummy. This case is basically solvable in engineering terms.
The second is data the Agent has to read and reason over itself. A financial-analysis Agent that isn’t given financial data simply can’t work. Here you can’t just substitute. One compromise is to run a tightly isolated sub-Agent that alone touches the sensitive data and hands back only a conclusion. But there’s an unavoidable catch: no one can guarantee the conclusion itself isn’t sensitive. Leaks tend to be paraphrased and buried in the meaning, where keyword matching won’t catch them. Isolation shrinks the blast radius; it doesn’t truly seal it.
The third and most thorough is information-flow tracking: label every piece of data, follow where it flows, and intercept it the moment it tries to leave. Microsoft’s Fides takes this route, using information-flow control to track the movement of every piece of data — the cleanest approach in principle. But it’s also the most expensive: it slows performance, hurts usability, the labels multiply, and someone has to write and maintain the policies. Between security and usability, this is where the trade-off bites hardest.
The systems engineering of agent security: isolation, control, and traceability
None of these approaches solves the problem on its own: guardrails get outflanked, sandboxes don’t stop data leaving, information-flow tracking costs too much, and sub-Agent isolation can’t guarantee the conclusion isn’t sensitive.
But that’s exactly how security has always worked. There’s no absolute “secure” or “insecure”; what you can do is keep raising the cost of an attack and shrinking the blast radius of an incident.
So for a team that genuinely wants to build an enterprise-grade (ToB) Agent product, there is no single silver bullet. The workable move is to stack three things. First, isolation: use system-level sandboxes and outside constraints to strictly bound what the Agent can do. Second, policy control: use least privilege and fixed rules to constrain what, within that bound, it’s actually allowed to touch. Third, traceability: use logs and an audit trail so that if something does go wrong, you can pin down which step and which party caused it.
The first two are prevention; the third is easy to forget and just as important. Guardrails are about stopping bad things from happening; logs are about being able to explain what happened once they do — and neither can substitute for the other. There’s a telling gap in practice: plenty of companies have reported suspected or confirmed Agent security incidents, yet very few actually treat the Agent as an entity that needs its own authorization and its own accountability. That gap is the most concrete weakness in Agent security today.
AI Agents are moving from “answering questions” to “doing the work for you.” The more autonomous the Agent, the more you’re handing over — decision-making, data, and every kind of credential and key. Hoping the model learns restraint on its own, or writing it a few hundred more guardrail rules, both stay inside the model, at the same level as the model, and can’t hold back the problems described above. For an enterprise to make an Agent genuinely usable, the line of defense has to be built outside the model: bound what it can operate on with a sandbox, constrain the data it can reach with least privilege, and log every step so that if something goes wrong it can be traced to the source. Those three layers together are what earns the name of an enterprise-ready Agent security architecture.