Gemma 4 E2B leads on H100

On a single H100 80GB, 8 small and medium models were compared using vLLM 0.19.1 and vllm bench serve. Each run was measured under 100 prompts, 128 in / 128 out tokens, and concurrency 1/4/8/16 conditions, with metrics of throughput(tok/s) and TTFT(ms).

At concurrency c=16, throughput was highest for Gemma 4 E2B-it 3180 tok/s, followed by Gemma 4 E4B-it 2015, Qwen 3.6 35B-A3B-FP8 1243, Gemma 4 26B-A4B-it 1033, Qwen 3.6 35B-A3B 718, Qwen 3.6 27B-FP8 557, Qwen 3.6 27B 439, and Gemma 4 31B-it 226, in that order.

There are three key conclusions.

• Small expert models overwhelmed large dense models. On the same GPU, Gemma 4 E2B showed 14x higher throughput and much lower latency than Gemma 31B dense.
• FP8 was especially effective for MoE. Qwen 35B-A3B recorded +73% throughput with FP8 applied compared to BF16, while Qwen 27B dense only saw a +27% improvement.
• 30B-class dense models were disadvantageous for single-H100 serving. Gemma 4 31B-it's TTFT spiked from 130ms to 4159ms as concurrency increased.

The author concluded that MoE-family models face significant HBM bandwidth bottlenecks, making FP8 a potential default choice, and that dense 27B/31B-class models are better suited for batch processing.