Google Unveils Next-Generation Open LLM Gemma 2

Google has released Gemma 2, its next-generation open LLM. The new model is available in 9B and 27B parameter sizes, each offered as a pretrained Base model and an Instruction-tuned model optimized for instruction following.

Gemma 2 was trained on about twice as much data as the previous generation. The 27B model leveraged 13 trillion (13T) tokens and the 9B model 8 trillion (8T) tokens of web data, code, and math data.

Key technical innovations include:

• Sliding window attention: Alternates between sliding window and full-quadratic attention to improve generation quality.
• Logit soft-capping: Prevents logits from growing excessively large, ensuring training stability.
• Knowledge Distillation: Transfers knowledge from the larger model to the smaller model (9B) to maximize performance.
• Model Merging: Applies techniques that combine multiple models to improve performance.

Gemma 2 is tightly integrated with the Hugging Face ecosystem, supporting usage via the Transformers library as well as fine-tuning with TRL and integration with Google Cloud.