New Benchmark Tests AI Agents on Complex Software Building
Replit has released ViBench 2.0, a rigorous new benchmark designed to test AI agents' ability to build complex, real-world software products from scratch. Unlike simple feature checks, this benchmark requires agents to build entire applications, such as those modeled after Uber or GitHub, feature by feature, without knowing the final product's requirements.
While the benchmark successfully tests individual components—with most top models passing single-feature checks—the results reveal a critical weakness: agents struggle significantly when features must interact or integrate. The failure points are found where different parts of the system meet, such as handling merge conflicts in code hosting or enabling cross-word commenting in document editors.
Performance metrics show that only a handful of advanced models can pass a significant percentage of the total build tests. The benchmark highlights that simply having powerful individual models is insufficient; true software development requires robust system-level reasoning and integration capabilities.
Why it matters
- —It moves beyond testing isolated features, focusing on the difficult challenge of system integration.
- —The results provide a clear, quantitative measure of an AI agent's ability to handle real-world software complexity.
- —It sets a new industry standard for evaluating the true capabilities of advanced AI models in development.
Key facts
- ViBench 2.0 tests agents on 17 products, covering 241 features and 544 test plans.
- Only three models were found to pass more than 40% of the total build tests.
- The primary failure point for top models is not individual features, but the integration where features meet.
- The benchmark shows that newer models, like GPT-6.1 Sol, are significantly improving performance over older versions.
The full text is in the original source. Here we provide a brief summary and key facts.