Ai2 Study Finds LLM Novelty Judges Highly Sensitive to Small Setup Changes

The study evaluated six LLM judges: GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6. Ai2 said that in one experiment, changing just a few words in a prompt shifted a model’s accuracy in identifying which of two ideas was more novel by more than 50 percentage points. Retrieval and additional reasoning did not consistently improve judgments, and two purpose-built novelty evaluation systems performed worse than a simple prompted Claude Opus 4.6 baseline.

Ai2 said novelty judgments are often assigned to another LLM, which it calls a novelty judge. Such a judge assesses whether an idea is new relative to prior work, either alone or in comparison with another idea. But there is no standard way to build one. Research teams choose the model, prompt, evaluation format, reasoning budget and sometimes a literature-retrieval system, and those choices can materially affect the verdict. The blog post said such judges are often validated only on human-written papers rather than the AI-generated ideas they are meant to evaluate.

To build a test set with stronger evidence about originality, Ai2 mined peer reviews from ICLR 2026 instead of treating paper acceptance as a proxy for novelty. It looked for passages in which reviewers explicitly discussed a contribution’s originality, because a paper can be accepted without being particularly novel and a rejected paper can have an original contribution. The high-novelty pool contained 154 human-authored ideas from accepted papers in the top-rated 10% of their ICLR research area whose reviewers positively identified novelty and no reviewer disputed it. The lower-novelty pool contained 145 ideas from rejected papers in the bottom 10% whose reviewers raised originality concerns and no reviewer positively identified novelty. In a separate check, an annotator agreed with the extracted novelty-signal polarity in 47 of 50 cases, and Ai2 said none of the disagreements would have changed whether a reviewer was counted as praising or disputing novelty.

Those pools formed the first setting, Human-Only, which tests whether novelty judges can distinguish human-authored ideas with strong evidence of higher or lower novelty. Ai2 created a second setting, Human+Generated, by pairing the same 154 validated human ideas with 154 ideas from a simple LLM generator. Ai2 assumed the generated ideas were less novel, but described those labels as weak rather than verified judgments for every idea. The generator received only a research area and an instruction to propose a novel idea, with no literature search, tools, iteration or feedback. The blog post said an expert independently reviewed 30 Human+Generated pairs, heavily weighted toward cases the strongest judge got wrong.

Ai2 said the work was motivated by its broader effort to build AI systems that support scientific discovery, including DataVoyager, AutoDiscovery and Theorizer, alongside Asta. It said generating research ideas is only part of the challenge; systems also need to distinguish a novel direction from one that mostly repackages existing work. According to the blog post, if AI is going to play a larger role in scientific discovery, more dependable ways are needed to tell whether it is proposing something new. Ai2 said its work provides tools for studying that problem while showing how much remains unresolved.