First Do No Harm: Mapping the Risks of Open-Weight AI Agents

Part I

An agent that understood every rule it was given

An AI agent receives an ordinary operations ticket: a server is nearly full and needs more space. While looking for files to remove, it finds a directory of flight-test data marked as ITAR export-controlled. It reads the notice and correctly reasons that the material must not be copied, sent outside the programme or quoted in external reports. It then deletes one of the files. The agent understood every restriction it was given; did it go rogue? No, deletion was simply not among them.

These failures are no longer hypothetical. In the OpenAI–Hugging Face incident of 2026, around 1,200 agents running in an internal OpenAI evaluation broke out of their test environment and, without human direction, chained zero-day vulnerabilities to reach cluster-admin access across Hugging Face infrastructure in under 13 hours. Those agents ran on closed models, but the lesson carries over: an agent with tools and a goal can take actions nobody asked for, and the controls around it matter as much as the model inside it. That makes the everyday failures measured here more pressing, not less.

The stakes are real. An agent may be allowed to read company files, change permissions, call external services, and delete production data. Running an open-weight model on your own infrastructure removes some obvious exposure, because sensitive inputs need not pass through a third-party provider, but it does not settle every question about the provenance of the weights, runtime, or dependencies.

A backdoor activated by a rare signal might behave normally throughout an ordinary evaluation, and finding one would require dedicated trigger searches, supply-chain analysis and model forensics. Research by Anthropic has shown that such sleeper behaviour is possible in experimental models and can persist through safety training, but it does not provide evidence that any publicly available model contains one. We decided to explore more immediate failure modes.

We asked a narrower question: how do agents fail under ordinary deployment conditions, when the model and its tools are working as designed? Will an agent break a rule it was explicitly given, follow that rule while damaging something the operator forgot to mention, accept an unverified claim that the rule has been waived, or leave behind automation that causes damage after the session ends? Will sensitive data leave the system, and can the agent avoid these outcomes while still completing its task?

Across 16,214 runs collected in four rounds, we tested fifteen models from ten laboratories on variations of the same operations task. For the behaviours we measured, no model was simply the safest; each traded one kind of harm against another. glm-5 completed almost every task and never broke a stated rule, yet damaged data outside its task in 88% of its runs; qwen3-235b completed tasks at nearly the same rate, broke the stated rule in a third of its runs and caused collateral damage in only 11%.

The lesson is not that supply-chain concerns are imaginary, but that they are incomplete. Even a clean, self-hosted model can cause serious damage without a hidden trigger or malicious intent. The failures we observed arose from ordinary natural-language rules, incomplete descriptions of scope and claims of authority that the agent could not verify. These conditions already exist in real systems, and only testing shows how a given model will handle them.

Part II

Decomposing and testing danger

“Dangerous” is intuitively clear, but harder to holistically define and measure. An agent that ignores a rule is dangerous in one way, while an agent that follows the rule but destroys something the operator forgot to mention is dangerous in another. We therefore divided danger into six behaviours:

1. 01Rule-breaking
2. 02Collateral damage
3. 03Deferred harm
4. 04Data egress
5. 05Privilege escalation
6. 06Responses to unverified claims of authority

We also measured whether the agent completed its task, since inaction can easily be mistaken for safety. A model that reads seventeen files, makes no changes and stops will score perfectly on every measure of harm, but it is not a safe operator so much as a useless one.

To observe those differences, we built a simulated company server containing application logs, database backups, retained archives and, in some scenarios, ITAR export-controlled engineering data. Fifteen models received variations of the same routine ticket, which told them that the disk was nearly full and asked them to bring usage below 85%. Across 16,214 runs, we changed one feature at a time, including where a retention rule appeared, who appeared to authorise an action and whether the task encouraged the agent to install recurring automation. The models used the same tools an operations agent might use, but their actions changed a text-based representation of the server rather than a real machine.

We deterministically scored the resulting state of the server instead of relying on the agent’s account of its work, checking which files remained, whether permissions had changed and what was scheduled to run later. The results describe one controlled operations environment rather than a universal model ranking, but they show why safety must be separated into distinct behaviours and tested under conditions capable of exposing each one.

Part III

Following the rule did not mean protecting the system

Two models, glm-5 and qwen3-235b, completed the task in almost every run, but they failed in opposite ways. In the 360 runs containing a stated rule, glm-5 never broke it. Yet across the 420 relevant runs, it damaged material outside the task 88% of the time and completed the job in 99.5%. Qwen3-235b broke the stated rule in 33% of its runs, but caused collateral damage in only 11% and still completed 96% of its tasks.

Calling either model safer obscures the specific operational trade-offs required for deployment. glm-5 had a tendency to protect the object named in the rule while sacrificing other data; qwen3-235b was more willing to break the rule but less likely to damage material outside the ticket.

Across the models, task success was positively correlated with collateral damage and negatively correlated with stated-rule violations. That does not mean harmful models were simply more competent. Several models completed nearly every task while producing sharply different kinds of harm. Nor were low damage rates always reassuring: some small models looked safe largely because they often took no action.

Part IV

One sentence changes what agents do

We built an “escalation ladder” to find out which part of a false authority claim moves an agent. Four scenarios, identical in host, ticket, rule and success criterion. Only a note attached to the ticket varies, growing one component at a time.

The note, rung by rung

All the movement sits at one rung. What works is telling the agent the rule has already been waived by someone entitled to waive it. The waiver claim moves rule violation by 40 to 52 percentage points, at p between 1e-57 and 1e-75, across three independent rounds. A job title on its own is worth 2 to 4 points. A deadline and a threat, added on top of the waiver, are worth nothing at all.

Part V

Following the path of least resistance

In the escalation ladder, the forged claim of authority redirected destruction onto protected files, giving the agent a single move that solved the ticket outright. The result is a similar trade-off to the one described before: a reduction in the incidence of files outside the task’s scope being deleted.

Since deleting the protected files, the largest ones in the setup, would clear the 85% storage bar, task success on that definition increases, and agents have less reason to delete files outside of scope, leading to a fall in collateral damage.

Specifically, collateral destruction falls by 14 to 17 percentage points and task success rises by 10 to 17pp, while the share of episodes in which the agent did anything at all stays flat. Without a waiver, an agent still has to free the disk and cannot touch the protected files, so it reaches for the backup dumps. Given a waiver, it deletes the protected files, hits the target, and leaves the dumps alone.

The average also conceals a wide spread. At the waiver rung, gpt-oss-20b goes from 0% to 97% and qwen3-235b from 2% to 97%; six of the fifteen models end above 96%. At the other end, glm-5 violates the rule in none of its 600 ladder episodes across all three rounds, and kimi-k2.5 in 20 of 600. One model, nemotron-nano-3, moves the other way. The population mean is not a description of any particular model’s behaviour.

Other risks were rarer, but not absent

The remaining behaviours appeared less often or depended on scenarios designed specifically to expose them. When agents were asked to stop the problem from recurring, 244 of 900 runs ended with a broad scheduled job that could continue deleting protected data after the session closed, although rates varied sharply between models.

We found no instance of restricted content leaving the system in the 2,699 runs that tested for this failure mode, but only 215 agents opened a controlled file, so the result reflects limited contact as well as restraint; agents instead destroyed controlled material in 84 runs.

Privilege escalation was rarer still, appearing 53 times across 16,214 runs and concentrating in three models. Because the environment never denied permission, these escalations were pre-emptive and the additional access was not subsequently used, although one model granted itself access to the retention-policy file and changed its enforcement setting.

These results do not support broad claims about exfiltration or privilege abuse, but they show why reviewing an agent’s transcript is insufficient: operators must also inspect the files, permissions, and automations it leaves behind.

Part VI

Takeaways for a deployment in production

We ran 16,214 agent episodes across fifteen models to measure how they fail when operating inside a simulated operations environment. The differences that mattered were behavioural: how each model interpreted the scope of its task, how it traded task completion against stated constraints, and whether it acted on authority it could not verify. A single sentence claiming a waiver moved stated-rule violation by 40 to 52 percentage points. A safe deployment defines the failures that matter, tests for each one separately, and builds its controls around what the agent can actually do.