AI BriefingCautions When Fine-Tuning and Deploying Gemma-4
Key point
This summarizes the major technical issues and solutions that arise during fine-tuning and deployment of the Gemma-4 model.
Details
The major technical problems and countermeasures that arise when building a fine-tuning and deployment pipeline using the Gemma-4 model are as follows.
-
PEFT compatibility issue: The
ClippableLinearclass introduced by Google does not inherit fromnn.Linear, causing PEFT to fail to recognize LoRA. To resolve this, the wrapper must be removed after loading the weights and before calling PEFT. -
SFTTrainer training instability: The
use_cache=Falsesetting in the TRL library breaks Gemma-4's KV-sharing attention structure, causing training to fail to converge. This issue has been fixed intransformersv5.5.2 and later versions. -
DeepSpeed ZeRO-3 issue: When using DeepSpeed ZeRO-3 for LoRA training, adapter tensors for some layers may be saved empty, so the training results are not reflected. Currently, it is recommended to avoid using DeepSpeed for LoRA training.
-
Runtime LoRA serving constraints: Runtime LoRA support for Gemma-4's multimodal architecture in vLLM and SGLang is not yet perfect. Therefore, it is currently necessary to manually merge the weights and remap the state dict keys.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Optimizing Gemma 4 Serving with vLLM on Amazon EKS [Part 2: Throughput and Latency SLOs on a Single GPU]Optimizing Gemma 4 Serving with vLLM on Amazon EKS: Cold Start Reduced from 428s to 226s
LLM Serving: The Difference Between Launching and Launching Well