Making LLMs Faster Without Sacrificing Accuracy

As LLM inference costs grow, Amazon researchers proposed an architecture framework at ICLR to find faster models while preserving accuracy. The core idea is to extend the Chinchilla scaling law beyond just parameter count and training tokens to include design variables as well.

Existing scaling laws couldn't explain why, even with the same parameter count and data, inference throughput can vary by up to 40% depending on architecture. The researchers added hidden size (d_model), the MLP-to-attention ratio (r_mlp/attn), and GQA as optimization variables, and separately calibrated how each affects loss and throughput.

• Increasing hidden size reduces FLOPs and KV cache within a fixed parameter budget, improving throughput.
• Adjusting the MLP-to-attention ratio reduces attention-side parameters and KV cache, easing memory bottlenecks.
• GQA compresses key-value heads to further lower generation costs.

However, bigger isn't always better for these three variables. Both hidden size and the MLP-to-attention ratio show U-shaped loss curves with an optimal point, and GQA showed high variability, so it was selected as a discrete hyperparameter via local search.

The framework works in two stages: first fitting the standard Chinchilla law to estimate coefficients E, A, B, α, β, then learning a correction surface for how each architectural choice affects loss. Through this process, Panda emerged as a model family aimed at maximizing accuracy, while Surefire targets the accuracy-efficiency Pareto optimum.

After training over 200 models ranging from 80M to 3B parameters and 8B to 100B tokens, Panda-1B achieved 2.1pp higher accuracy than LLaMA-3.2-1B but had 33% lower throughput. In contrast, Surefire-1B maintained nearly the same accuracy while being 21% faster on vLLM and 47% faster on SGLang.

Panda-3B achieved 0.6pp higher accuracy than LLaMA-3.2-3B but had 23% lower throughput, while Surefire-3B maintained accuracy while boosting throughput by 12–17%. The authors concluded that the optimal MLP-to-attention ratio for LLaMA-3.2-style models is about 1.0, much lower than the 4.8 used in existing public models.

• Small-scale experiments were able to predict the optimal architecture for large-scale models.
• The efficiency gains were consistent across A100 and H200 GPUs, and across vLLM and SGLang.
• Depending on conditions, throughput improved by up to 42% on A100 (vLLM) and up to 47% on H200 (SGLang).