AI4Bharat and NVIDIA Release 2.8 Trillion Token Dataset for 22 Indian Languages

AI4Bharat, in collaboration with NVIDIA and IIT Madras, has unveiled Nemotron-pretraining-Indic-v1, a monumental resource comprising approximately 2.8 trillion tokens. This dataset is specifically designed for pretraining large language models (LLMs) across 22 major Indian languages, which represent the official languages recognized in India's Constitution. This release is not merely a dataset; it is a comprehensive package that includes an open, GPU-accelerated data generation pipeline, providing model builders with the complete 'recipe' used to create the massive corpus. The sheer scale and scope of this effort are designed to address a critical technological gap: the historical underrepresentation of Indian languages in the global AI data ecosystem, which has largely been dominated by English data.

The core innovation lies in the open pipeline itself. Developers are not just given the data; they receive the full workflow, including prompts, LLM operational workflows, and rigorous quality assurance checks. This allows the community to adapt the system to their own unique source materials and generate specialized data tailored to specific needs. AI4Bharat, the dedicated Indian language research lab at IIT Madras, has long been building open-source resources, such as the Sangraha corpus, laying the groundwork for open Indic language research. This joint release represents a massive expansion of synthetic Indic training data, making advanced model development significantly more accessible to the global community.

To enhance the technical depth of the models, the corpus incorporates a broad mix of curated English material—including specialized content like mathematics, programming code, and question-answer pairs—which is then translated or adapted into the target Indian languages. This ensures that models trained on this data can not only understand conversational language but can also explain complex technical concepts, solve structured problems, and answer detailed questions in these languages. Crucially, every piece of synthetic text generated is traceable back to a curated source document, and its output is checked against its input, ensuring that the generated text remains factually grounded in real-world information. This commitment to provenance is vital for building trustworthy and reliable AI systems.

Addressing the foundational challenge, Mitesh Khapra, head of AI4Bharat, highlighted that the primary bottleneck in Indian language technology is the scarcity of sufficient high-quality data compared to English. This release directly tackles that problem by providing not only massive quantities of data but also the means to generate more. The corpus is meticulously structured to account for the diverse ways people write in India. It includes native-script data for all 22 languages, alongside romanized text (written using the Latin alphabet) for twelve of them. This results in a complex structure spanning 34 distinct language-script configurations. The data generation workflow is sophisticated, combining document-level translation, quality assessment, and targeted repair mechanisms. Furthermore, the system includes specific checks to preserve the integrity of specialized content formats, such as equations, tables, and code blocks, which are often lost during simple translation.

This extensive data resource complements the naturally authored Indic text, which remains irreplaceable for capturing local knowledge and unique cultural context. The technical architecture of the corpus is detailed, noting that the token counts describe the retained corpus, including accepted rewrites, rather than the total token budget of a single training run, and that these counts are dependent on the specific tokenizer used (in this case, the Gemma-4 tokenizer). The corpus is built using documents sourced from 15 different Nemotron-CC-v2 subsets, covering diverse domains like general web text, Q&A, math, and code. NVIDIA's NeMo Data Designer framework is utilized throughout the workflow to manage the translation, quality checks, and targeted repair processes. The ability to trace every output back to its original source document is a key feature that developers can leverage. The release also details how Nemotron 3.5 Super VL was further enhanced by incorporating 10 billion Indic tokens, drawn from a twelve-language subset of this corpus combined with natural Indic text, demonstrating the immediate practical application of the new data resource. By opening up both the massive data resource and the entire generation process, the goal is to democratize large-scale Indic data generation, enabling a wider range of developers to build robust and sophisticated AI models across the linguistic diversity of India.

The distinction between native-script and romanized data is crucial for model builders. While both can express the same underlying content, they offer different script exposure to the model. For instance, the corpus includes native-script data for languages like Assamese, Bodo, Dogri, Kashmiri, Konkani, Maithili, Manipuri, Sanskrit, Santali, and Sindhi, while the romanization covers a subset of accepted translations for the twelve dual-script languages. Understanding this distinction allows developers to construct a highly nuanced training mixture, maximizing the model's exposure to various linguistic representations while maintaining content integrity. This level of technical detail and open methodology sets a new standard for multilingual AI development in underrepresented language groups, providing a powerful blueprint for similar efforts globally.