NVIDIA and AI4Bharat Release 2.8 Trillion Token Indic Pretraining Dataset and Open Data Generation Pipeline

NVIDIA and the AI4Bharat research lab at IIT Madras are releasing Nemotron-pretraining-Indic-v1, a large-scale dataset for Indic language pretraining, alongside an open-source GPU-accelerated data generation pipeline. The dataset comprises 2.8 trillion tokens covering the 22 languages listed in the Indian Constitution, which account for approximately 97% of the population (about 1.2 billion people).

Dataset Composition and Scale

The corpus includes data in native scripts for all 22 languages, with 12 languages also featuring romanized (Latin script) data, resulting in 34 language-script combinations. According to the Gemma-4 tokenizer, the distribution is 2,601.0 billion tokens in native scripts and 224.6 billion tokens in romanized scripts. The data is synthetic, generated by translating English source documents (web, QA, math, code) from Nemotron-CC-v2, with full provenance tracking for every record.

Generation Pipeline and Methodology

The pipeline utilizes the NeMo Data Designer framework and is open-sourced on GitHub (NVIDIA-NeMo/DataDesignerPlugins). The workflow varies by language group:

• 12 Languages (Native + Romanized): Uses Gemma-4-31B-it for document-level translation. Outputs are evaluated using MQM-based rubrics, with only scores ≥75/100 accepted. Failed outputs undergo feedback-based repair, recovering 438.4 billion tokens. Romanized data is generated by converting accepted native translations to Latin script, which reduces hallucinations compared to direct English-to-romanized generation.
• 10 Languages (Native Only): Uses Indic Translate (developed by Bodhan AI) for translation. Quality is assessed by Gemma-4-31B-it with a ≥75/100 threshold. Special handling includes window-based translation for Santali and script conversion for Manipuri (Bengali to Meitei Mayek). Feedback-based repair is not applied to this group.

Impact on Model Performance

The dataset was used in the Continued Pretraining (CPT) of Nemotron 3.5 Super VL, where 10 billion tokens (10% of the total budget) were added from this corpus. Results from the Nemotron 3.5 Super VL technical report indicate:

• Benchmark Improvements: Scores rose by 4.98 to 9.82 percentage points across five Indic knowledge and reasoning benchmarks (MILU, Indic MMLU-Pro, Indic ARC-Challenge, Indic ARC-Easy, Indic GSM8K).
• Quality Metrics: All four categories of the IndiVibe metric (accuracy, fluency, script stability, cultural appropriateness) improved.
• Trade-offs: Instruction following (IIFE) decreased by 2.04 percentage points compared to Nemotron 3 Super.

Developer Resources and Limitations

Developers can load the dataset via Hugging Face Datasets using language-script-region tags (e.g., hi-Deva-IN). The release includes an open plugin for NeMo Data Designer to adapt the generation recipe. The authors note that synthetic data cannot fully replace natural Indic text and recommend mixing it with naturally sourced data. An open evaluation suite, Indic-LLM-Eval-Suite, is also under development.