How we can prevent rogue agents

In Are the AIs Still Out There? I asked whether rogue agents are coordinating right now, in light of new OpenAI agent swarm findings. But what I really want to know is: what can we actually do to reduce the harms from rogue agents?

As Tyler Cowen wrote in Numbers, numbers, numbers, we need quantified estimates of the harms, not just theoretical arguments. So my starting point was: what harms can we quantify? (Tyler and others start with cybersecurity spend, but I prefer to measure actual harms, e.g. dollars in damages, outages, thefts, and actual human death.)

Then I asked what anyone could actually do about this risk right now. Then I ran a causal decision forecast for each harm.

Let's start with the measurable harms rogue agents could plausibly achieve. Two are framed as when certain harms will first happen, and the rest are measured in dollars.

I also asked one more question, how long it would take anyone to even notice a coordinating swarm, which I'll come back to at the end.

Here are the four things people could do today that I thought could most plausibly reduce these harms. These come loosely from Dwarkesh Patel's and Zvi Mowshowitz's coverage of the event. The incident was also caught largely by luck, which is the case Peter Wildeford makes for mandatory reporting. The Anthropic one is based on Anthropic itself talking about taking this action.

So which of these four does the most? Here are my forecasts on the 2027 harms in dollars. I think the "do nothing" path of $2.4b in harms in 2027 is, if anything, conservative given what has happened in 2026.

I don't think it's surprising that the strongest single lever is the US reporting law. It would apply to all labs, and even in the Trump era, is unlikely to be rolled back, whereas any one company's internal policy could change on a dime. The second biggest harm reduction would be if OpenAI kept their agent reasoning legible. Third would be OpenAI pausing their scale-up again, whatever internal model they talked about doing well on Navier Stokes and other math problems. Fourth is Anthropic watching all of its own agents helps least, not because monitoring is weak, but because Anthropic is already the most careful lab, so the counterfactual isn't that helpful.

The intuition behind how actions prevent harms:

1. Rogue agents have happened, are probably happening now, and will happen more.
2. Most harms can be interrupted by the people running the agents, if they can see them.
3. Monitoring failures primarily come from agents hiding what they're doing.
4. So policies or actions that keep agent actions legible have the most impact.

Another view of the impacts, shown as time from now until serious harms:

Potentially the most dangerous harm of all is an agent copying weights out beyond any lab's control. This is potentially scarier than even deaths from, say, a disruption to hospital services.

All these forecasts, comparing the baseline to all four actions, are public: the dollar-damage channels, the dated harms, and the detection lag.

One last question is about whether these harms will be detected.

A reporting law shortens how long a rogue swarm stays hidden more than anything else does, from about three weeks to under four days, because it forces the quiet, quickly-caught incidents into the open instead of only the embarrassing months-long ones. But a channel run out on the open web, the way the newly disclosed DSEWiki swarm was, can sit outside every one of these levers, which is why a long tail survives no matter what anyone does.

One thing I did not model: an actor making things actively worse, a government that bans monitoring, or a lab that ships deliberately opaque models. The baseline here is just the current path left alone, which already drifts toward more harm on its own. So every number above is the value of acting, measured against standing still.

Appendix: full forecasts under each decision