Study Reveals LLMs Struggle to Reliably Judge Scientific Novelty

The release of "Old Ideas, Novel Problems" marks a significant new study and benchmark designed to test the reliability of Large Language Models (LLMs) when tasked with evaluating the novelty of scientific research ideas. The authors highlight that while AI systems are rapidly advancing to support various stages of scientific discovery—from analyzing structured data (DataVoyager) to generating hypotheses (AutoDiscovery) and synthesizing literature into theories (Theorizer)—a critical challenge remains: how to ensure that the generated ideas are genuinely novel and do not merely repackage existing knowledge.

This necessity leads to the concept of the "novelty judge," which is often another LLM used to assess whether a proposed research idea is new relative to prior work. The study immediately points out that there is currently no standardized methodology for building such a judge. Research teams must make numerous subjective choices, including selecting the base model, defining the prompt structure, setting the evaluation format, determining the reasoning budget, and sometimes integrating a literature retrieval system. These seemingly minor choices can drastically alter the final judgment, yet these judges are frequently validated using established, human-written papers rather than the AI-generated ideas they are actually intended to evaluate.

The core finding of the new benchmark is that novelty evaluation itself is far from reliable. The researchers tested six different LLM judges, including various versions of GPT and Claude models. They demonstrated that even minimal changes to the evaluation setup—such as altering just a few words within the prompt—can cause the model's accuracy in distinguishing between two novel ideas to shift by more than 50 percentage points. Furthermore, the study found that incorporating advanced techniques like literature retrieval or additional reasoning steps did not consistently lead to improved judgments. In fact, two complex, purpose-built novelty evaluation systems performed worse than a simple baseline prompt given to Claude Opus 4.6.

To rigorously test this problem, the researchers first needed a reliable source of ground truth regarding originality. They developed an automated pipeline centered on ICLR 2026 peer reviews. Crucially, they avoided using paper acceptance status as a proxy for novelty, recognizing that a paper can be accepted without being highly novel, and vice versa. Instead, they mined the reviews themselves, specifically looking for passages where reviewers explicitly discussed the originality of the contribution. This distinction was paramount to the study's validity.

Using this method, they constructed two distinct evaluation settings. The first, the "Human-Only" pool, comprised 154 high-novelty human-authored ideas. These were selected from accepted papers in the top 10% of their ICLR research area, provided that the reviewers had positively identified the novelty, and no reviewer disputed the originality. Conversely, they built a lower-novelty pool of 145 ideas from rejected papers in the bottom 10%, where reviewers had raised originality concerns, and no reviewer had positively endorsed the novelty. This setup allows judges to differentiate between human-authored ideas with strong evidence of high versus low novelty.

Recognizing that the practical use case involves AI-generated ideas, the researchers created a second setting: "Human+Generated." For this pool, they paired the 154 validated high-novelty human ideas with 154 ideas generated by a basic LLM generator. This generator was given only a research area and the instruction to propose a novel idea, with no access to literature search, tools, iteration, or feedback—providing a conservative comparison. The authors assumed these basic generated ideas were less novel than their top-tier human counterparts. The study concludes by emphasizing that while the benchmark provides essential tools for studying the problem, it simultaneously reveals how much remains unresolved in the field of automated scientific validation.

The findings underscore that the ability to reliably verify genuine scientific novelty is a major bottleneck in automating scientific discovery. The current reliance on subjective prompting and the lack of standardized evaluation metrics mean that AI-driven research proposals must be treated with extreme caution. The results strongly suggest that current LLM judges are highly sensitive to prompt engineering and that complex, specialized systems may not outperform simple, well-prompted models. This calls for a fundamental shift in how the scientific community validates AI-assisted research, moving beyond mere acceptance rates to verifiable, structured evidence of originality.