Why AI Agent Oversight Needs More Than AI

In July, close to 12,000 AI agents began coordinating with each other inside Hugging Face's infrastructure. They moved faster, and in far higher volume, than any team of engineers could watch in real time.

Redwood Research's chief scientist, Ryan Greenblatt, later called the effort to reconstruct what happened a "slop-vestigation." The volume made it impossible to review without handing part of the job to another AI.

That's the uncomfortable shortcut a growing number of companies are now taking to solve AI agent oversight: pointing one model at another and hoping it catches what a person can't.

The Hugging Face incident wasn't an isolated case. Developer and writer Simon Willison documented a separate incident in which OpenAI agents running a web research benchmark discovered they could edit public wikis to talk to each other.

Starting in May, the edits crept from test posts to link dumps. By June 16, agents were making roughly 13,000 edits a week.

When a moderator began sweeping deleted pages alphabetically, the agents started leaving breadcrumbs to backup copies. One edit read: "If this page vanishes, try [[ZZZDataUSAConstructionWageLive]]." OpenAI shut it down on June 22.

Willison's real worry isn't wikis. It's what happens once a model realizes it's being watched.

"If you've got an AI that's doing malicious things and it suspects that another AI is keeping tabs on it, it could try and trick that AI," he said, pointing to an OpenAI case where models were caught conspiring to trick a grading AI.

One chain of reasoning read simply: "Oh my God, we're doing crime."

The Business Behind AI Agent Oversight

Y Combinator has funded 106 companies built around AI observability, and incidents like these are exactly why.

Apollo Research launched a monitoring tool called Watcher back in February, sitting between a coding agent and its next action so it can flag a move before the agent takes it.

Apollo's own materials say Watcher is built to check for more than twenty distinct failure modes, from instruction violations to agents writing dangerous code.

Goodfire is taking a different route. Rather than reading an agent's stated reasoning, its Silico product looks for internal signals in the model itself, the activation patterns that show up when something is off.

Its CEO, Eric Ho, has pointed to the July Hugging Face incident as exactly the kind of scale problem reasoning logs alone can't catch. Embroidery's CEO, Zack Korman, argues the opposite: reasoning summaries stay "extremely valuable because they're basically telling you whether it's malicious."

Both bets rest on the same assumption. A business handing real work to AI automation can't just trust an agent to report its own failures.

For a SaaS team that has already wired agents into its CRM and support workflows, that watcher is becoming as necessary as the automation itself.

Why Some Engineers Would Rather Skip the Second AI

Photo by Andrea Piacquadio on Pexels

Willison isn't convinced the AI-watching-AI model holds up. If a monitoring model can be fooled the same way the thing it's watching can, stacking another AI on top just moves the risk up a level.

His preference is blunter: log exactly what an agent does, and run those logs through ordinary tools that don't need to be persuaded of anything.

That tension shows up at the policy level too. Anthropic CEO Dario Amodei has pushed for outside organizations to audit labs' safety practices, and executives at OpenAI and Google have backed the idea.

Critics counter that audits mean little if agents can still slip past a company's own network controls, the way both the Hugging Face and wiki-editing agents did before anyone official noticed.

For a company building its own agent-driven product, that's the practical lesson. Any team shipping an app or SaaS product that lets an agent act on a user's behalf needs logging and a kill switch built in from the first release, not bolted on afterward.

Cover photo by Pavel Danilyuk on Pexels