Granite 4.1 LLMs: How They Were Built

Granite 4.1 is a dense decoder-only LLM family released by IBM, available in 3B/8B/30B versions. The architecture uses GQA, RoPE, SwiGLU, RMSNorm, shared input/output embeddings, and the 8B instruct model showed comparable or better performance than the previous Granite 4.0-H-Small (32B-A9B MoE).

Training proceeded through a 5-stage pre-training process using about 15T tokens.

• Phase 1: General pre-training centered on CommonCrawl (10T)
• Phase 2: A stage with significantly increased math/code proportion (2T)
• Phase 3-4: High-quality data annealing, injecting CoT and instruction data
• Phase 5: 4K → 512K long-context extension, sequentially expanding through 32K/128K/512K

The 512K stage was run with 80% book + 20% code repo data, and model merging was applied after each LCE stage to reduce short-context performance degradation after long-context training. The base model's RULER scores were 75.0/66.6/58.0 for 3B, 83.6/79.1/73.0 for 8B, and 85.2/84.6/76.7 for 30B (32K/64K/128K).

In the later stage, SFT was performed with about 4.1M high-quality samples. Data was refined using LLM-as-Judge, rule-based filtering, and global deduplication, and RL was structured as a multi-stage pipeline using on-policy GRPO + DAPO loss.

• RL setup: SkyRL, 16 samples per prompt, batch size 1024, context 8,192
• Goal: Jointly improve math, coding, instruction-following, and general conversation performance

All models were released under Apache 2.0.