Gewell Inference Engine Optimized for Gemma 4 31B Released

The inference engine Gewell, optimized to maximize the architectural characteristics of the Gemma 4 31B model, has been released. By implementing Gemma's lossless KV cache reduction capabilities—which were not utilized by existing solutions like vLLM or llama.cpp—it is optimized for Blackwell GPUs and large-scale concurrent workloads.

Core Technology: KV Cache VRAM Reduction

Gemma features a structure where K and V weights are tied in the Global attention layer, and RoPE rotates only 25% of K. Gewell leverages this characteristic to store only 25% of V and K instead of the full K and V, reducing VRAM usage to 0.625x. This approach improves memory efficiency without mathematical loss, and given Blackwell's high compute capacity and lower VRAM bandwidth characteristics, the reduction in memory reads outweighs the increase in computational load.

Cache Management and Performance Characteristics

While vLLM's LRU policy is inefficient for workloads with repetitive prompts, Gewell supports explicit cache hints and smart eviction policies. For example, in workloads sharing history, such as Writer-Critic loops, it prevents unnecessary evictions and keeps chat sessions warm. For designed workloads (multiple growing prompts that are not simultaneously active), overall throughput (t/s) is higher than vLLM, though Time To First Token (TTFT) for large batches may be slightly slower.

Quantization and Limitations

It uses a proprietary mixed precision quant format, with the default setting (G0) using approximately 6bpw. Most bits are allocated to Attention and Global-attention-adjacent MLPs to maintain quality. Current limitations on the interface side include rudimentary sampler support (temp, top-k, top-p) and the inability to override chat templates. Windows and RTX 5090 environments have not been verified.