vLLM Introduces Hardware-Agnostic Layer Architecture to Support Diverse Accelerators

vLLM is introducing a new hardware-agnostic (HW agnostic) layer set to resolve compatibility issues between its new "flat" model definitions and torch.compile. This change aims to support out-of-tree (OOT) accelerators, older GPUs, and unique model architectures without sacrificing the performance gains achieved by recent optimizations for NVIDIA Blackwell and other cutting-edge hardware.

The Conflict: Performance vs. Portability

Recent vLLM updates moved toward "flat" model definitions that bypass torch.compile to achieve state-of-the-art performance on specific hardware. While effective for NVIDIA GPUs, this approach breaks compatibility with OOT plugins (e.g., IBM Spyre, Huawei Ascend) and older consumer GPUs that rely on torch.compile for optimization. This created a maintenance burden where supporting new models required separate implementations for every hardware backend.

The Solution: HW Agnostic Layers

The new architecture adheres to four design principles:

• Compilable: Model definitions remain compatible with full-graph torch.compile.
• Extensible: Retains CustomOp and PluggableLayer mechanisms for OOT plugin overrides.
• Isolated: Separates hardware-specific paths from general model logic to allow independent development.
• Portable: Implements layers using native PyTorch or portable DSLs (Triton, Helion).

Performance and Implementation Status

On NVIDIA H100 GPUs, the HW agnostic layers achieve a total token throughput within 3.4% of native implementations (geometric mean across three recent models). In some cases, they even slightly outperform native models due to efficient portable implementations.

• Current Status: Limited layers are merged into the main branch (PR #49458).
• Activation: Users can enable the feature via the USE_HW_AGNOSTIC=1 environment variable when using the transformers backend.
• Validation: Tested successfully with Gemma 4, Qwen3, and Granite 4.2 on IBM Spyre OOT plugins.
• Future Plans: CI integration is underway, with plans to make this the default serving path for Spyre and provide HW agnostic model.py files for all flat models.