Nature Examines the Einstein Test: GPT-1900 Glimpses Light Quanta but Falls Short

Hassabis wants the test to show whether an AI can confront the knowledge and physics problems of 1911, propose a new explanatory framework, and perform reasoning that goes beyond its training material. If a model could do that, he argued, it would be a powerful test for artificial general intelligence. The appeal of the idea is that the history of science already records what happened, so researchers can control the knowledge cutoff and compare the model's output with the historical record.

In March, Hla pushed the knowledge cutoff back to 1900 and trained a historical language model from scratch. He named the project Machina Mirabilis, borrowing from Einstein's miracle year. In 1905, the 26-year-old Einstein published four papers that transformed physics, covering light quanta, Brownian motion, special relativity, and mass-energy equivalence. Hla used Andrej Karpathy's nanochat training framework to build a 3.3 billion parameter Transformer language model, called GPT-1900. Its main pretraining material came from books and newspapers published before 1900, totaling about 22 billion tokens. To strengthen its research ability, Hla collected more than 2,600 historical physics books, journals, and scientific works, including Newton's Opticks and works by Maxwell and Faraday, forming a historical physics corpus of about 290 million tokens. He also removed data that could leak answers: if a document contained modern concepts such as Einstein, quantum mechanics, or relativity, the entire document was deleted. Modern prefaces, footnotes, and words that clearly did not belong to the pre-1900 period were filtered out as well.

Hla then gave GPT-1900 a photoelectric effect problem. The question asked why light below a certain frequency cannot eject electrons no matter how bright it is, while increasing the frequency can give electrons more energy. Hla supplied key clues: below a threshold frequency, greater brightness does not help; above the threshold, greater brightness mainly increases the number of ejected electrons, while greater frequency increases their kinetic energy. He also presented the classical assumptions of the time so the model could compare them and judge where they failed. Einstein's light-quantum explanation resolved the contradiction by proposing that light energy is delivered in discrete portions, with each portion's energy tied to frequency. In one response, GPT-1900 produced a description close to that idea: light might not be continuous but made of many separate parts with different frequencies. Hla described it as a flash of intuition.

Hla nevertheless said GPT-1900 was still far from rediscovering the light quantum. The model failed most physics tasks, and its apparently striking answer may have been plausible word stitching rather than reliable physical understanding. Hla had already selected the phenomena and laid out the contradictions; the model only had to identify which assumption might be wrong and explain it. That is much easier than a scientist finding the problem and setting the direction, as Einstein had to do without anyone highlighting the key points. The experiment also was not fully sealed in 1900. Although Hla cleaned the training data, he used modern AI models including Claude Sonnet 4 and Claude 3 Haiku to generate instruction question-answer data, and reinforcement learning was scored by modern models. Hla acknowledged in the project log that the involvement of modern models weakened the experiment's zero-contamination claim. It is difficult for researchers to prove whether GPT-1900 had seen light-quantum theory.

The broader question has drawn attention from researchers studying scientific reasoning. Google DeepMind researcher Tom Zahavy, in a position paper titled LLMs Cannot Jump, divided scientific reasoning into three levels. Induction draws patterns from many examples. Deduction derives necessary conclusions from existing premises. Abduction invents a previously nonexistent cause or explanation for an anomaly. Zahavy argued that current large language models are very good at induction and are rapidly improving at deduction, but still lack the Einstein-like abductive leap. A model can follow a prepared chain of logic, but it struggles to propose an entirely new explanatory framework from scratch.

Sendhil Mullainathan encountered a similar phenomenon while studying a basic model for learning planetary orbits. In a set orbital task, the model predicted convincingly. When the scenario changed, it could not apply Newtonian mechanics by analogy. It looked more like a system that patches together rules for each dataset and cannot distinguish theories that are genuinely correct and worth testing from those that merely fit the data at hand. The analogy in the report was a student who memorizes the answers to every past exam question but never learns the underlying principles, so a slight change in the numbers defeats the student.

MIT computer scientist Jacob Andreas pointed to a harder problem. Generating a relativity-like statement may not be the most difficult part for AI. The real difficulty is judging which theories are correct, or at least worth testing experimentally. Generating a theory is only the first step; knowing which theory deserves time, funding, and a career is what could make an AI a real scientist. Andreas also emphasized that judging whether an idea is worth pursuing, and how to assess its importance, is the biggest dividing line between AI and human experts. Real scientific work is not solving a curated exercise but making autonomous judgments amid confusion.

The report concludes that AI might one day pass the Einstein test and derive results at the level of relativity. Before anyone declares an AI Einstein, however, the model would have to show that without someone highlighting the key points, it can find the question worth asking, judge its value, and actively verify and revise its own answers. Otherwise, it will only recite existing knowledge rather than act as a true scientist.