Fast Gemma's Proven Inference Optimization Recipe (7 min read)

The VIDRAFT team optimized the inference speed of google/gemma-4-E4B-it on a single NVIDIA A10G in the Fast Gemma Challenge. The final verified result was 510.58 TPS with PPL 2.3930, marking one of the fastest results among those that passed the quality bar.

Looking at raw TPS alone, there was a run that recorded 535.91, but it was not verified since its PPL exceeded 2.44, failing the quality bar. The VIDRAFT team explained that they focused on verified results that maintained quality rather than raw speed.

The published manifest.json contains the full reproducible environment. The key configuration is as follows.

• Uses vLLM 0.22.1rc1.dev307, Transformers 5.9.0
• Applies sliding_window=188 and a FlashAttention sliding-window patch
• Sets CENTROID_TOP_K=49
• Uses MTP-based speculative decoding with 7 speculative tokens
• Applies LM head pruning and an int4 keep-set
• Uses a synthetic warmup bridge with 64 prompts and 1 token
• Enables CUDA graph, fused sparse argmax, and Split-KV verification
• Applies end-only detokenization and fast rendering

Running it requires serve.py, related patch files, the warmup bridge, the drafter, and weight buckets. GPU memory utilization was set to 0.90, max batched tokens to 512, and max number of sequences to 1, and temperature 0, top-p 1, and top-k 0 were used for consistency of generation results.