OpenAI audit finds major flaws in SWE-Bench Pro

OpenAI reports that a detailed audit of SWE-Bench Pro, a coding benchmark meant to measure realistic software engineering ability, found widespread problems in the dataset. The company estimates that about 30% of the tasks are broken, meaning the benchmark may not reliably separate genuine model limitations from flaws in the evaluation itself. This matters because benchmark scores are used to judge whether AI systems are improving, whether they are safe to deploy, and where research teams should focus next.

SWE-Bench Pro was introduced as a stronger successor to SWE-bench Verified, which OpenAI had previously criticized for design and contamination problems. The newer benchmark was designed to test longer, more realistic coding tasks, especially the kind of multi-step work expected from coding agents. Its tasks are built from feature changes in public and private software repositories. A model must write code that passes new tests for a feature while preserving existing behavior. On the public split of 731 tasks, frontier models reportedly improved from 23.3% pass rate to 80.3% over eight months, which made the benchmark an important signal for agentic coding progress.

OpenAI’s new audit suggests that this signal is much noisier than expected. The company used a data-point analysis pipeline that looked at model attempts, task metadata, and failure traces to identify likely evaluation flaws. Those flagged tasks were then examined through multiple investigator-agent passes and independently reviewed by five experienced software engineers, with disagreements escalated for more investigation. The automated pipeline identified 200 broken tasks, or 27.4% of the public split, while the human annotation campaign identified 249 tasks, or 34.1%. OpenAI therefore estimates that roughly one-third of SWE-Bench Pro tasks may be invalid or misleading as measurements of coding ability.

The audit grouped the main problems into four categories. Some tasks had overly strict tests that demanded a particular implementation detail even when the prompt did not specify it, so a functionally correct answer could be marked wrong. Other tasks had underspecified prompts, where hidden tests enforced requirements that a model could not reasonably infer from the visible instructions. A third group had low-coverage tests, meaning the benchmark did not check the requested feature well enough and incomplete fixes could pass. Finally, some prompts were misleading because they pointed models toward behavior that contradicted what the tests actually required.

OpenAI describes the goal of the audit as making sure that failures and successes mean what they appear to mean. A failed task should indicate a real model weakness, not a bad prompt or unfair hidden test. A passed task should mean the model produced a complete and valid solution, not that the tests were too weak. To check this, OpenAI built a quality assurance process that first used an automated filter to inspect instructions, model submissions, and grading tests. That filter flagged 286 potentially broken tasks, which were then reviewed more deeply through both agent-assisted analysis and a human engineering review.

The agent-assisted review used Codex-based investigator agents with access to the task repository and environment. These agents could inspect files, run tests, study repository conventions, and compare common model failure patterns. This was intended to distinguish fair ambiguity from true underspecification. In real software projects, not every requirement is written explicitly, and engineers often infer behavior from nearby code. The audit tried to separate that normal kind of context-dependent work from tasks where the benchmark was genuinely asking for something the prompt did not support. After several independent investigator runs, a researcher reviewed the summaries and made the final issue labels.

In parallel, OpenAI ran a human annotation campaign using experienced software engineers trained on the benchmark’s goals, issue taxonomy, and edge cases. Each flagged task was reviewed by five engineers. Reviewers considered the visible problem statement, test cases, and the reference solution, also called the gold patch, before using the pipeline analysis or transcripts as supporting context. This combination of automated triage, agent investigation, and human review was designed to make the audit more scalable while still grounding final judgments in professional software engineering expertise.

The broader lesson is that hard coding benchmarks are difficult to curate because realism and fairness can pull in different directions. Real-world software tasks often require context, conventions, and judgment, but a benchmark must still define success clearly enough that correct solutions are not rejected and incomplete solutions are not rewarded. As models become stronger and benchmark scores rise quickly, small flaws in task design can have large effects on how progress is interpreted. OpenAI advises model developers to examine SWE-Bench Pro results carefully rather than treating headline pass rates as a clean measurement of software development capability.

The audit also points to a growing role for AI agents in checking evaluation data. OpenAI used agents not as the final authority, but as scalable investigators that could inspect repositories, run tests, and summarize likely flaws for human review. That suggests a practical path for improving benchmarks: use agents to find suspicious data points at scale, then use expert reviewers to make final calls. For AI safety and deployment decisions, the key issue is not only how well models perform, but whether the tests used to measure them are trustworthy.