llama.cpp Adds SarvamMoE Support
Key point
A PR has been submitted to llama.cpp adding support for the Sarvam-specific MoE architecture.
Details
llama.cpp PR #20275 adds a dedicated sarvam_moe path.
- Extends the structure for Sarvam-30B based on
BailingMoeForCausalLM. - 19-layer configuration: 1 dense FFN + 18 MoE layers, 128 routed experts, top-6, 1 shared expert.
- Reflects sigmoid gating, zero-mean expert bias normalization, and
routed_scaling_factor=2.5. - Also includes GQA (64 heads, 4 KV heads), combined QKV projection, and QK RMSNorm.
- The tokenizer handles converting SentencePiece BPE to GPT-2 byte-level encoding.
While the official Sarvam GGUF can also be loaded via the existing bailingmoe2 path, this PR focuses on adding a dedicated Sarvam architecture and conversion logic to correctly align RoPE, expert bias, and tokenizer handling. Testing confirmed HF safetensors conversion, multiple quantizations (Q8_0/Q6_K/Q4_K_M), and inference behavior. However, a comment on May 8th pointed out issues with Indic script and English tokenizer consistency, and the maintainer requested a rebase onto master.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.