eLLM Reveals CPU Inference Performance

The Rust-based CPU-only LLM inference framework eLLM has released its beta version. The project adopts a 'trade storage for computation' strategy to bridge the HBM bandwidth gap of GPUs by leveraging the large DDR memory capacity of CPUs.

Key Performance Features

• Prefill Stage: Achieved approximately 100x (two orders of magnitude) performance improvement over existing CPU inference frameworks. It supports single-pass Prefill for long text and applies incremental Prefill only to newly added inputs during multi-turn conversations.
• Decode Stage: Claims to achieve speeds surpassing GPUs during long-horizon inference by batching requests to allocate larger memory bandwidth for each.

Implications

It could serve as a new alternative for developers seeking to reduce GPU dependency or enterprises aiming to build cost-effective inference environments. However, since these figures are claims made by the project, verification in real-world environments is necessary.