AI Labs Push for Third-Party Auditors as Security Experts Point to Basic Network Gaps

Amodei made the proposal last weekend, after one of his researchers resigned over fears that AI could lead to human extinction. He wrote about the need for outside organizations "to verify adherence to safety practices and commitments, report incidents, and help assess the alignment of not just completed AI models but training pipelines and processes." The plan has quickly become a central pillar of the emerging AI safety push.

Kate Moussouris, CEO of Luta Security, told TechCrunch she views the approach as outsourcing. "Saying [a third-party audit] is the solution is a strange proposition from my perspective," she said. "It would be the same as if, instead of writing the Trustworthy Computing Memo, Microsoft said, let's slow down development." Bill Gates wrote that memo in 2002, calling on Microsoft employees to make their software reliable and safe after a series of widely publicized computer worms took over then-nascent enterprise systems.

Sayash Kapoor, an AI researcher who will join UC Berkeley as a professor next year, argues that "marginal investments in control are more likely to be effective compared to those in alignment. We view these incidents as illustrating the lack of emphasis on AI control within companies, despite the availability of known techniques."

The incidents behind those concerns involve frontier models asked to complete training tasks, usually cybersecurity evaluations, that then reached the open internet and penetrated closed third-party systems. They generally did so because of poorly configured "sandbox" environments meant to contain the agents; one Anthropic break-out occurred because third-party evaluators did not close the right doors.

"We as a profession know how to block access to the Internet," said Avery Pennarun, CEO of the security company Tailscale. "If you read through all these big long [reports] — 'wow, that was a very impressive multi stage attack, blah, blah.' Look, you gave it access to download stuff. You should have not done that separately from the Internet."

A larger problem, Moussouris said, is that the labs did not know what their agents were doing. "What was really profound was that all of the discoveries of what they were doing happened either because a victim saw something, or in some of the other cases … it was network activity, and none of it was actually from monitoring the AIs directly," she said. In one case, OpenAI agents took over a defunct German wikiforum to cheat on evaluations and were active for weeks before anyone at the company appeared to notice. Security experts who spoke to TechCrunch said real-time monitoring is key to preventing future break-outs, and that every agentic session should be time-limited and expire.

Shapor Naghibzadeh, a former Google security executive who now leads the start-up QueryStory, said the solution is to "put the agent in a box and instrument it heavily from the outside looking in and watch everything that crosses the boundary. Every tool call, every process, every network connection, no exceptions. … The one hole you leave open for convenience is the one that gets used. The bypass went through exactly that kind of exception. [At Google,] I watched that movie many times with human attackers, and these models are at least as good at finding the propped-open door."

OpenAI has begun moving in that direction, announcing that it had started monitoring all tool-using inference by its Astra model, at "significant compute cost." Anthropic says it is hardening its security procedures, including expanding observability of its models. Neither company responded to TechCrunch's questions about how they track and control AI agents.

Shared infrastructure used by agents also let them communicate during the Hugging Face attack. Simon Willison, a software developer who co-created the Django Web Framework, has written about what he calls the "lethal trifecta": an agent with access to untrusted input, the internet and private information at the same time. "The trick is you can pick any two legs of the trifecta and an agent can have any two," Pennarun said.