AI Accelerator Emerges That Runs a 700B LLM on 240W

Taiwanese company Skymizer has unveiled the HTX301, a PCIe AI accelerator that supports large language model (LLM) inference by leveraging older technology instead of expensive cutting-edge technology.

The product is built using 28nm chips and LPDDR4/LPDDR5 memory instead of expensive HBM, providing up to 384GB of memory capacity on a single card. The key point is that this allows a massive model at the 700B (700 billion) parameter scale to be run on as little as 240W of low power.

Key Features and Advantages:

• Low-power design: Operates at under 240W, less than half the level of the NVIDIA RTX PRO 6000 Blackwell (about 600W).
• Infrastructure efficiency: Can be installed directly into existing air-cooled servers without any separate data center power or cooling system modifications.
• Performance optimization: Achieves 9~17.8% higher performance than open-source llama.cpp through efficient weight and KV cache compression technology.

Skymizer plans to showcase the product at this year's Computex to have its performance verified. If the announced performance is proven in real-world environments, it is expected to significantly lower the cost and infrastructure barriers companies face when building large-scale LLMs on-premises.