LayerStoRm: Consumer GPU LLM Serving

The open-source engine LayerStoRm has been released to serve frontier-scale MoE LLMs in environments with limited VRAM. The tool operates by leveraging RAM and parallel PCIe transfers to fetch only the experts actually routed for each token, utilizing pre-loaded VRAM as a smart cache.

Performance and Configuration

The test environment consists of 2x RTX 5090, 2x RTX 5080, and 512GB RAM. When running the GLM 5.2 UD-Q4_K_XL model, it achieved approximately 10 generation tokens per second and 44 prompt processing tokens per second. This is significantly higher than the 1.37 gen tok/sec recorded when using llama-bench on the same configuration.

Key Technical Features

• Custom Transfer Scheduling: Applies a micro-optimization solver that iteratively solves for optimal transfer settings per token-layer
• KV Cache Management: Supports HiSparse KV offloading, TurboQuant or SnapMLA KV compression, and PagedAttention
• Hardware Optimization: NUMA optimization, Pinned RAM, and low-latency transfers via DMA waterline queues
• Serving Features: OpenAI-compatible API, tool calling, streaming, and KV cache and model weight sharding when TP > 1

Currently, it only supports GLM 5.2, DeepSeek V4, and DeepSeek 3.2 architectures, and is limited to Linux and Nvidia RTX 5080-5090 series GPUs. It is released under the MIT license, with future support for AMD and hybrid configurations planned.