LlamaIndex unveils BooBench to stress-test document parsers
LlamaIndex has released BooBench, a challenging new benchmark designed to rigorously test the capabilities of document parsers. This benchmark consists of 179 pages, which are drawn from real-world documents and are specifically engineered to push parsers into failure modes, such as repetition loops or incorrect data extraction.
The pages mimic complex document structures, including tables filled with zeros, near-identical rows, and rare scripts. The core goal of BooBench is to assess a parser's ability to detect and avoid hallucinated or repeated information. A parser fails (gets a "Boo") if it repeats data beyond what is present on the page, returns nothing, or times out.
The results reveal significant performance disparities: LlamaParse Agentic and Agentic Plus achieved a perfect score (179/179), while the best frontier VLMs scored 99.4%, and open VLMs scored between 61% and 93%. This underscores the critical need for standardized, robust testing when deploying document understanding AI in enterprise settings.
Why it matters
- —It exposes critical weaknesses in current document parsing models, especially regarding data repetition and hallucination.
- —It provides a standardized, highly rigorous test suite for enterprise-level document AI.
- —The performance gap shown between proprietary and open models highlights the current state of the art in document understanding.
Key facts
- BooBench is a benchmark of 179 pages using real-world, complex documents.
- The benchmark specifically targets failure modes like repetition loops and data hallucination.
- LlamaParse Agentic and Agentic Plus achieved a perfect 100% score on the test.
- Open VLMs scored between 61% and 93%, indicating variability in performance.
The full text is in the original source. Here we provide a brief summary and key facts.