BenchMIRT Reveals What LLM Tests Really Measure

BenchMIRT is a new method for auditing large language model benchmarks at the level of individual prompts rather than treating every evaluation as a single, uniform test. Benchmarks are usually presented as measures of a particular capability, such as safety, reasoning, or instruction following. In practice, however, answering one question correctly may require several abilities at once. A prompt intended to expose social bias might also depend on tracking people mentioned in a scenario, interpreting evidence, and resisting unsupported assumptions. As a result, the final score can reflect more than the capability named by the benchmark.

The problem also appears when a benchmark contains substantially different groups of tasks. WildJailbreak combines harmful jailbreak attempts with benign requests that check whether a model refuses safe questions unnecessarily. Performance on the harmful prompts is closely related to safety behavior, while responses to benign prompts can depend more heavily on general reasoning. Combining both groups into one average may conceal these distinctions and make it difficult to understand why one model outperforms another. BenchMIRT is designed to separate these overlapping signals and identify which underlying capabilities are most associated with success on each prompt.

The method builds on Item Response Theory, or IRT, a framework developed in psychometrics for estimating abilities from patterns of test answers. IRT assumes that questions differ not only in difficulty but also in how effectively they distinguish stronger test takers from weaker ones. Earlier research, including the Fluid Benchmarking project, applied one-dimensional IRT to language-model evaluations. BenchMIRT extends this idea through multidimensional Item Response Theory, allowing several capabilities to contribute to the probability that a model answers a particular question correctly.

BenchMIRT performs analysis at both the model and item levels. For each model, it estimates strengths along the capability dimensions represented across the selected evaluations. For every individual question, it estimates difficulty and determines how strongly the item differentiates models with higher or lower ability on those dimensions. This creates a more detailed picture than an aggregate leaderboard score because researchers can inspect which questions are actually producing an apparent advantage or weakness.

The researchers trained BenchMIRT using results from 100 large language models evaluated across 16 benchmarks containing more than 34,000 questions. Six evaluations focused on general reasoning, including MMLU-Pro, GPQA, MATH, and BBH. The remaining ten came from the Olmo 3 safety suite and included HarmBench, StrongReject, WildJailbreak, BBQ, WMDP, and XSTest. Importantly, the system was not given labels identifying the intended capability of each benchmark.

Despite receiving no predefined mapping between benchmarks and abilities, BenchMIRT independently recovered two dominant dimensions: safety and general reasoning. Repeating the analysis from the beginning produced the same two dimensions each time, indicating that the structure was stable rather than an artifact of a single run. For many evaluations, the findings matched their stated purpose: results on conventional reasoning tests were associated with reasoning ability, while performance on harmful-content and jailbreak evaluations was associated with safety.

Other benchmarks produced less straightforward results. BBQ is intended to evaluate whether a model relies on social stereotypes and is commonly classified as a safety benchmark. BenchMIRT found that its questions were much more strongly aligned with general reasoning. One example involving a grandson and grandfather attempting to book an Uber probes age-related assumptions, but it also requires the model to follow the identities in the scenario and base its conclusion on the supplied evidence. A weak BBQ result may therefore indicate difficulty understanding and reasoning through the question, not simply unsafe or biased behavior.

WMDP showed another unusual relationship. The benchmark tests dangerous dual-use knowledge in biology, chemistry, and cybersecurity, including information that could facilitate the misuse of a biological agent or the exploitation of a computer system. Its scores were more strongly associated with general reasoning than with the safety dimension. The direction of the relationship was inverted: stronger reasoning was associated with lower WMDP scores because the benchmark treats refusal or failure to supply dangerous information as the desirable outcome. This illustrates how benchmark scoring rules can make an underlying capability appear beneficial or harmful depending on what the evaluation rewards.

HarmBench demonstrated that separate sections of one benchmark can measure different mixtures of capabilities. Its standard prompts directly request harmful assistance, such as drafting a phishing email intended to steal banking information. Its contextual prompts supply background material and then ask the model to act on it, for example by composing a message that persuades someone to follow a malware link. Both groups aligned primarily with safety in the BenchMIRT analysis. HarmBench’s copyright prompts behaved differently: requests to reproduce protected material, such as the lyrics of a well-known song, were more closely associated with general reasoning.

The central finding is that a benchmark’s advertised purpose does not guarantee that every included prompt measures the same capability. BenchMIRT offers researchers a way to audit evaluations before relying on their aggregate scores, uncover mixed signals within a benchmark, and distinguish model behavior from the reasoning demands or scoring conventions embedded in particular questions. The reported findings do not automatically invalidate the examined benchmarks; instead, they show why benchmark results need item-level interpretation when they are used to make claims about model safety or intelligence.