AI BriefingAMD MI300X Optimized Kernels Released
Key point
Hugging Face and AMD collaborated to release custom kernels specifically for MI300X that maximize the performance of Llama 3.1 405B.
Details
Hugging Face and AMD collaborated to develop open-source custom kernels to optimize the inference performance of the Llama 3.1 405B model on AMD MI300X GPUs.
This optimization aims to improve performance when using FP8 precision in a vLLM environment, and includes the following three core kernels:
- Fused residual connection, RMS norm, and FP8 conversion kernel
- Fused SwiGLU activation and FP8 conversion kernel
- Skinny GEMM kernel
Using these kernels in combination can significantly reduce the decoding latency of the Llama 3.1 405B model on MI300X-based nodes.
The developed kernels have been released via the hf-rocm-kernels GitHub repository, allowing users to perform their own benchmarking or build their own kernels using them. These kernels are planned to be integrated into the vLLM project for AMD in the future.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
vLLM Publishes Performance Benchmarks for 5 Speculative Decoding Methods on AMD GPUs
GLM-5.3-Flash Builds Inference Infrastructure on Cluster of Over 100,000 Chinese Accelerators, Achieving ~3x Performance Boost