Nick Cammarata Reverses 'Cat-Car Neuron' Finding, Says Neural Networks Are Already Clean

The 4e:55 neuron became famous because it fired strongly for both cat faces and car fronts. Cammarata now says the neuron does not recognize cats or cars; it detects V-shaped geometry. Fox snouts, flames, car fronts and V-shaped fireboat sprays activate its positive pole, while round faces, balls and wheels activate its negative pole. He attributes the original misreading to crude observation tools.

In 2023, Cammarata worked on OpenAI's first large-scale automated neuron explanation project, where existing tools explained only about 2 percent of neurons. Three years later, with more advanced algorithms and agent teams, he says modern vision models' final layers yield clear explanations for 99 to 100 percent of neurons. Even InceptionV1, long known for deep-layer confusion, became much more interpretable when reexamined with the new tools. He defines 'clean' not as one neuron doing one thing, but as a highly coherent and precise internal organization that humans can read directly, like a map that can be zoomed into.

OpenAI core researcher roon amplified the article, calling it an important moment in the history of science and saying it could make AI alignment a 'normal science.' If the argument holds, it would undercut a core anxiety of the past decade: that interpretability must first disentangle features inside a messy network. According to the Leiphone report, the article suggests that sparse autoencoder decomposition, feature dictionaries and feature disentanglement may have been pursuing a problem created partly by the observation tools themselves.

The decade-old consensus grew out of InceptionV1, a 2014 vision model used as an early 'model organism' for interpretability. Researchers found surface neurons relatively clear but deeper layers increasingly tangled, and estimated that only about 5 percent of final-layer neurons were understandable. The 4e:55 neuron, which responded to both cats and cars, appeared to violate the expectation of one neuron per concept. The superposition hypothesis explained this by saying networks fold unrelated concepts into the same neuron when capacity is limited, because those concepts rarely activate together. In the 2020 Zoom In paper, Cammarata and Chris Olah added evidence that car features were hidden inside a group of dog neurons. That led labs including Anthropic and OpenAI to rely on sparse autoencoders to pull entangled features apart and label them for humans.

Cammarata began to suspect the problem was not the network but the tools. He traced the shift from feature visualization first developed around 2009, which produced hallucinatory images, to transformed robustness and clearer visualizations, to his team's current method: fine-tuning regularization parameters so algorithms follow real images' natural activation paths. In modern architectures such as ConvNeXt, he says, more than 99 percent of final-layer neurons became clearly readable. The past decade, in his account, was spent fighting tool error.

Two canonical ambiguity cases were revised. In the cat-car case, the old feature visualization used gradient descent to synthesize the image that most strongly activated a neuron, but the algorithm took shortcuts, creating a reward-hacking effect. Because ImageNet contains many strong cat and dog images, the strongest synthetic activations carried semantic residue such as cat whiskers or dog fur, leading researchers to read the neuron as an animal detector. His team used activation-path guidance and cross-validation of positive and negative poles. The neuron, they concluded, classifies by geometry: V-shaped contours on one end, circular contours on the other.

In the car-dog case, InceptionV1's training data had far more dogs than cars, and dog-related neurons were denser, so optimization fell into a dog-image basin and produced dog-heavy strongest activations. Using automated agent tools, gradient tracing, ablation experiments and generalization-axis disentanglement, Cammarata's team found the circuit was not a dog neuron but a detector for objects facing the camera directly and symmetrically. Car features received a 'front-facing' tag there and continued downstream, eventually splitting into separate car and dog classification circuits. The earlier ambiguity, they argue, came from dataset bias and from features naturally generalizing and branching along signal pathways.

Cammarata explains polysemanticity through recursive generalization. A neuron may start by recognizing rapeseed flowers; along a yellow-feature axis it extends to yellow objects, along a round-shape axis to spherical objects, then to dogs in flower fields, and finally to green broccoli. Seeing only the endpoints makes the neuron look confused, but tracing its growth reveals coherent branching at each step. In language models, he says, a neuron that initially detects the 'iy' sound can extend to 'diy' and even to the phrase 'Do it yourself,' which does not contain that sound.

To test whether neurons switch to superposition when capacity is tight, Cammarata focused on InceptionV1's 3b layer, which has no residual stream for signals to bypass. He compressed 480 curve-detecting neurons into 10, a 48-fold reduction, and retrained from scratch. Under superposition theory, the network should cram unrelated features into those 10 neurons. Instead, he reports, all 10 remained clearly interpretable, though each covered more ground: one curve-and-loop neuron merged 10 angle-specific curve detectors, and two circle neurons absorbed multiple circular categories. He also found that features disentangled by SAE were not cleaner; their response widths were several times coarser than those of original neurons, which he says raises the possibility that some claimed superposed features were mathematical artifacts produced by SAE itself. Later, however, he acknowledges that in normal, uncompressed original layers, SAE and native neuron shapes match at 0.96 to 1.00, so SAE remains a useful tool.

Google DeepMind mechanistic interpretability team lead Neel Nanda, a co-founder of superposition theory, pushed back. He said the article is interesting but brings no substantive breakthrough for interpretability. Nanda argued the two sides are not debating the same superposition: Cammarata attacks 'arbitrary superposition,' the idea that networks put unrelated features in one neuron, while Nanda defends 'representation superposition,' in which large language models use linear combinations of few neurons to store far more knowledge than neuron count. Nanda also said weak activations contain vast stores of facts and memories and are central to superposition; that the strongest evidence comes from vision models while language models differ in information density; and that SAE remains irreplaceable on residual streams, reflecting each layer's information bottleneck. Zapier co-founder Mike Knoop asked whether network cleanliness could become a formal evaluation metric; Cammarata said it would not be difficult and that control activation, ablation and blind-spot probing could score the underlying order.

Cammarata agreed with parts of the criticism and described his article as a working hypothesis. He listed unresolved issues: interpretability has not saturated as models grow, and the engineering challenges of extreme scale remain far from solved; the study covers only vision and language single-modality models, not vision-language fusion layers; recursive generalization was fully traced only in image models because language models lack training checkpoints; SAE was not wholly rejected; and the conclusions rely heavily on steering-oriented experiments and attribution graphs, which are perturbation-based methods.