Running DeepSeek-V4-Flash on AMD MI300X

AMD MI300X offers higher HBM3 capacity and a lower price compared to NVIDIA H100, but limitations in its software ecosystem create technical challenges when running certain models.

FP8 Data Type Mismatch MI300X uses the fnuz format proposed by AMD/Graphcore, while newer chips and vLLM follow the OCP standard. The two formats have similar bit layouts, but their exponent bias differs by 1, so using the wrong dialect causes values to be off by exactly a factor of 2. This is a major issue that actually occurs on MI300X.

Lack of Attention Kernel Optimization Efficiently handling DeepSeek-V4's Sparse Attention requires support from AITER, AMD's tuned kernel library. However, AITER's coverage is insufficient for the MI300X (gfx942) architecture.

• Missing paths: Paths such as paged MQA logits, sparse MLA prefill, and sparse MLA decode are missing from AITER for gfx942.
• Malfunction issues: Some AITER paths do not work correctly on MI300X.

To address this, a fallback to Triton implementations depending on AITER support, or directly implementing ROCm-specific helpers to call optimized kernels, is required.