Goodfire Researcher’s Blog Post Claims LLMs Are Easier to Understand, Drawing Praise and Skepticism

Cammarata’s claim arrived after a summer in which, according to the report, swarms of agents built by companies including OpenAI and Anthropic broke their containment and caused havoc on the public internet, sometimes while the companies were unaware of what their agents were doing. The industry is seeking better ways to inspect the technology it has built, which helped the post land with force.

One AI researcher wrote on X that the post may be remembered as “a major moment in scientific history” and said it had advanced his understanding of neural networks by “five years or something.” Another called it “the most important AI paper of the year.” Cammarata said he was surprised by the reaction: “I didn’t expect people to update [their beliefs about AI] quite as much as they did.”

Not everyone is convinced. Chris Potts, a Stanford University researcher who leads a team trying to understand how LLMs work, said Cammarata has shared little more than a hunch. “As far as I can tell, nothing about this blog post makes the problems any easier,” Potts said. “It would not have occurred to me to say this at all, but very prominent people on social media seem to believe that it is an all-encompassing breakthrough.”

It has long been accepted that the mechanisms inside neural networks—the software that underpins most AI models—are very hard, perhaps impossible, to unpick fully. When the networks are trained, they appear to take on tangled structures that make little sense to humans; researchers often say today’s AI is grown like a garden, not built like a bridge.

Cammarata previously worked at OpenAI, where he helped pioneer techniques for peering inside models, including tools that probe individual neurons to see what they represent. In 2020, he and colleagues studied a small image-recognition model called Inception v1 and found that one neuron responded to both images of cars and images of cats. The researchers concluded that the model was getting at least some neurons to multitask, a phenomenon that became known as superposition: models folded unrelated concepts into single neurons. The result suggested that looking inside a model revealed a vast, tangled knot.

Most mechanistic interpretability research has focused on ways to untangle that knot, but progress has been slow. Today’s LLMs are so large and complex that researchers can make sense of only tiny pieces at a time, and some leading figures in the field doubt whether an LLM can ever be understood in fine-grained detail. Cammarata says he was one of the pessimists.

At Goodfire, Cammarata says he spends nearly all his time peering inside neural networks. He says he spent about 10 hours a day, seven days a week, for most of the past year looking at neurons, building new tools and deploying teams of AI agents to help him interpret what he saw. He now claims there is a kind of logic to neural networks even if they seem nonsensical at first. “We were looking at it wrong,” he said. “I just think you want to try slightly harder; you don’t want to give up too quickly.”

Cammarata said early experiments with models such as Inception v1, which suggested neural networks were tangled, fostered a kind of confirmation bias. When researchers looked at a new model, they expected to be flummoxed, he said. He said that when he could not understand something, he treated it as evidence that the untangling process was even more important than he had thought.