JetBrains' Mellum 2

Mellum 2 is a 12B-parameter Mixture-of-Experts (MoE) language model specialized for software engineering tasks. It performs a variety of programming assistance functions including code generation and editing, debugging, multistep reasoning, tool use, and agentic coding, and is the successor to the previous 4B dense model.

The model architecture adopts an MoE structure that activates 8 of 64 experts. It combines Grouped-Query Attention (4 KV heads) with Sliding Window Attention applied to 3 of every 4 layers, and uses a Multi-Token Prediction head to achieve auxiliary training objectives while also serving as a draft model for Speculative Decoding to improve inference efficiency.

The model underwent 3-stage curriculum training using approximately 10.6 trillion tokens. Training progressively shifts from web data to refined code and math content, using Muon optimization and FP8 hybrid precision. It also achieves a 128K context window through YaRN technology.

Two versions of the model have been released: an Instruct model that answers directly, and a Thinking model that goes through an explicit reasoning process. Mellum 2 shows performance competitive with open-weight models in the 4B–14B range, while maintaining actual compute usage at the level of a 2.5B dense model, boasting high efficiency.