ViDoRe V3: Setting a New Standard for Enterprise Document Retrieval

In an era where enterprises handle increasingly complex, visually-rich documents, accurate information retrieval has become a critical challenge. Addressing this, ILLUIN Technology, in collaboration with NVIDIA, has released ViDoRe V3, a state-of-the-art benchmark designed to redefine the evaluation of multi-modal document retrieval for enterprise use-cases. Unlike traditional benchmarks that rely on clean, academic, or synthetically generated data, ViDoRe V3 emphasizes real-world enterprise relevance, diverse query types, and human-verified annotations. With its unprecedented scale and rigor, this benchmark promises to push the boundaries of retrieval-augmented generation (RAG) systems.

ViDoRe V3

ViDoRe V3 is the latest iteration in a series of benchmarks aimed at evaluating Visual Language Model (VLM) retrievers. It improves upon ViDoRe V1 and V2 by addressing core limitations: small corpora, reliance on synthetic data, and lack of human verification. V3 introduces a benchmark built from 10 enterprise-relevant datasets, 8 of which are public, encompassing domains from finance and pharmaceuticals to industrial technical documents and physics lectures. It contains 26,000 pages and 3,099 queries, each translated into six languages, with queries linked to human-verified relevance annotations, bounding boxes for key elements, and comprehensive reference answers.

The benchmark prioritizes enterprise realism: datasets are curated from multi-modal documents like text, tables, charts, images, and infographics. Queries are deliberately diverse—spanning seven types (open-ended, multi-hop, numerical, boolean, extractive, enumerative, compare-contrast) and three formats (question, instruction, keyword)—to capture complex retrieval scenarios across languages including English, French, Spanish, German, Italian, and Portuguese.

A key differentiator of ViDoRe V3 is its hybrid generation process. Queries are initially synthesized at scale using advanced LLMs like Qwen3-235B and NVIDIA’s NeMo Data Designer, then refined and validated by expert human annotators. Ground-truth annotation involved a multi-stage pipeline: VLM-based pre-filtering followed by meticulous human verification, multi-annotator consensus checks, and rigorous quality control using additional VLM validation to ensure precision and reliability.

Evaluation on ViDoRe V3 reveals its difficulty for current retriever models. Even the best-performing model, nemo-colembed-3b, achieves only 65% NDCG@10 on English datasets, and multilingual challenges reduce performance below 60% on average. Industrial and energy-related technical documents pose significant difficulty, while computer science texts show relatively higher performance, likely due to model exposure to coding data. Query-type analysis further highlights that open-ended and multi-hop queries remain the hardest, whereas extractive and boolean queries are more easily retrieved.

ViDoRe V3 also acknowledges limitations: coverage is focused on French and English, and while annotation quality is exceptionally high, some errors may remain. Despite these constraints, it sets a new benchmark for evaluating enterprise document retrieval, particularly in real-world, multi-modal, and multilingual contexts.

What Undercode Say:

ViDoRe V3 represents a paradigm shift in evaluating enterprise document retrieval. Traditional benchmarks often overestimate model capabilities by relying on synthetic or academically curated data that rarely reflects the chaotic complexity of real-world enterprise documentation. By integrating human verification, multi-modal data, and multilingual queries, ViDoRe V3 offers a more accurate, nuanced lens for evaluating retrieval systems. This is particularly important for enterprise RAG pipelines, where failure to retrieve critical information can translate into financial loss, regulatory risks, or operational delays.

One of the most impressive aspects of ViDoRe V3 is the scale and diversity of its datasets. Publicly available datasets cover finance, pharmaceuticals, HR, energy, industrial orders, and physics, each with unique multi-modal elements. The private datasets—covering energy regulations and telecom standards—allow for rigorous evaluation without the risk of model overfitting. This combination ensures that ViDoRe V3 is not just a benchmark but a stress test for VLMs in enterprise scenarios, highlighting both strengths and weaknesses.

From a technical perspective, the benchmark’s hybrid query generation and annotation strategy is ingenious. By leveraging LLMs for scale and humans for precision, the benchmark avoids the common pitfalls of synthetic-only datasets while remaining feasible to implement at a large scale. The multi-layered verification framework, with consensus checks and multiple stages of annotation, makes it one of the most reliable benchmarks currently available, minimizing biases and annotation errors.

Performance insights from ViDoRe V3 are equally revealing. The gap between technical and general-domain documents underscores the current limitations of VLMs in understanding domain-specific schematics, charts, and tables. The relatively strong performance in computer science documents suggests that model pretraining biases—like exposure to coding material—can skew evaluations and highlight uneven generalization capabilities. Moreover, the benchmark’s multilingual dimension exposes persistent weaknesses, reinforcing the need for cross-lingual capabilities in enterprise AI applications.

In essence, ViDoRe V3 provides more than evaluation metrics; it delivers actionable insights for model developers. For instance, improvements in multi-hop reasoning, cross-lingual comprehension, and domain-specific visual understanding are clear priorities for advancing enterprise retrieval. Additionally, the benchmark emphasizes that achieving high NDCG scores is not merely a matter of model scale but also strategic dataset design, annotation quality, and query complexity.

ViDoRe V3 is likely to become a standard tool for research and industry alike, offering a benchmark that aligns closely with enterprise needs. Its emphasis on realism, multilingual coverage, and multi-modal complexity will force developers to confront gaps in current VLMs and create models that are truly enterprise-ready.

Fact Checker Results:

✅ Dataset diversity spans 10 enterprise domains with 26,000 pages and 3,099 queries.
✅ Human-verified annotations include relevance rankings, bounding boxes, and reference answers.
❌ Current benchmarks still underperform on multi-hop, open-ended, and multilingual queries, indicating persistent gaps in model capabilities.

Prediction:

📈 ViDoRe V3 will drive the next wave of VLM improvements, with emphasis on multi-modal reasoning, cross-lingual retrieval, and domain-specific knowledge. Expect models specifically fine-tuned on enterprise data to close the performance gap within the next 12–18 months. Multilingual and technical document understanding will become key differentiators for leading retriever systems.

If you want, I can also create a visually-optimized infographic summarizing ViDoRe V3 datasets, query types, and model performance for easier sharing and readability. It would make the article even more engaging. Do you want me to do that?