Gemma 4 QAT Models: Optimizing Model Compression for Mobile and Laptop Efficiency

Google has released new checkpoints with Quantization-Aware Training (QAT) technology applied to maximize the efficiency of Gemma 4. This allows users to run the model locally on common edge devices and consumer GPUs.

While the conventional Post-Training Quantization (PTQ) method is prone to performance degradation, QAT simulates quantization during the training process to minimize the quality loss that occurs during model compression.

This release includes the popular Q4_0 format as well as a new quantization format specialized for mobile environments. In particular, using the mobile-specialized format can reduce the memory footprint of the Gemma 4 E2B model to under 1GB.

The following technologies were applied to optimize for mobile devices:

• Static activations: Data scaling calculations are performed in advance during training to reduce the workload on mobile chips
• Channel-wise quantization: Data is structured to match mobile accelerator designs, improving computation speed
• Targeted 2-bit quantization: The token generation portion is aggressively compressed to 2-bit, while core inference layers maintain higher precision
• Embedding and KV cache optimization: Focuses on compressing the vocabulary list and short-term memory to reduce active memory usage

These models can be downloaded via Hugging Face, and are provided in GGUF format for llama.cpp and as compressed tensors for vLLM.