Hot Chips 2026: High Bandwidth Flash (HBF) Application

High Bandwidth Flash (HBF), presented at Hot Chips 2026, implements SSD flash memory technology using packaging similar to HBM, aiming to resolve DRAM capacity shortages in LLM workloads.

Technical Characteristics and Limitations of HBF

• Unlike HBM, HBF requires access in large aligned chunks similar to SSDs, making byte-level random access impossible.
• Software must move data via DMA and perform SSD controller functions such as wear leveling and data retention, meaning it is not plug-and-play.

LLM Inference Application Scenarios

• A method is proposed in the vLLM runtime to store MoE (Expert) experts in HBF and DMA only active experts to HBM.
• KV cache can be stored in HBF to improve capacity efficiency if the sparse attention implementation reads only a subset of tokens; however, software optimization is required due to HBF's preference for sequential reads.
• In multi-GPU environments, a strategy to replicate model weights in HBF to increase local access is considered to reduce cross-device communication costs.

Cost and Software Challenges

• Although HBF has a lower cost per capacity than HBM, its cost per bandwidth is higher, making it suitable for small models or batch sizes that do not reach bandwidth limits.
• Adapting existing DRAM-based frameworks for HBF requires massive software refactoring similar to handling low-level disk APIs, with some noting that the complexity may be higher than simply streaming from an SSD.