StarCoder2 and The Stack v2 Released

BigCode has released StarCoder2, a next-generation open-source code LLM family, along with The Stack v2, a large-scale code dataset.

StarCoder2 is available in three parameter sizes: 3B, 7B, and 15B. In particular, StarCoder2-15B was trained on over 600 programming languages and more than 4 trillion tokens, achieving best-in-class performance for its size class and matching the performance of models with 33B parameters or more.

The training entities for each model are as follows:

• StarCoder2-3B: ServiceNow
• StarCoder2-7B: Hugging Face
• StarCoder2-15B: NVIDIA (using NVIDIA NeMo)

Technically, the models adopt Grouped Query Attention, a 16,384-token context window (with a 4,096-token sliding window applied), and Fill-in-the-Middle training.

The dataset, The Stack v2, is based on the Software Heritage archive and boasts a massive size of 32.1TB after deduplication. It features improved language and license detection and filtering algorithms compared to the previous version (v1), and is designed to group data by repository so that models can more effectively learn the context of code.