Gemma 4 QAT Models: Optimizing Model Compression for Mobile and Laptop Efficiency

Google has released new checkpoints applying QAT (Quantization-Aware Training) technology to run Gemma 4 more efficiently on consumer-grade GPUs and edge devices.

Unlike the traditional post-training quantization (PTQ) approach, QAT integrates the quantization process into the training process, minimizing the quality degradation that occurs during model compression. This release includes the popular Q4_0 format as well as a new quantization format specialized for mobile environments.

In particular, the following technical designs were applied to optimize for mobile devices:

• Static activations: Data scaling values are pre-computed during the training stage, reducing the computational burden on mobile chips and increasing response speed
• Channel-wise quantization: A data structure tailored to mobile accelerator design that supports native operations
• Targeted 2-bit quantization: The token generation portion is heavily compressed to 2-bit, while core inference layers maintain higher precision, achieving both performance and capacity simultaneously
• Embedding and KV cache optimization: Compression is focused on the vocabulary list and short-term memory, lowering memory footprint even during long conversations

Through these optimizations, the Gemma 4 E2B text-only model can reduce memory usage to under 1GB. Model weights are now available for download via Hugging Face in various formats, including GGUF (for llama.cpp) and compressed tensors for vLLM.