New llama.cpp LoRA Kernels Boost Strix Halo Inference Speed from 300 to 1400 t/s

·2026.10.21 14:24

Key point

Custom kernels reduce the performance penalty of applying 35 MB LoRA adapters to Qwen3.8-Flash-Next, raising throughput from 300 to 1400 t/s on Strix Halo hardware.

Details

New general and fused LoRA kernels for llama.cpp significantly improve inference speed when using LoRA adapters on Strix Halo hardware.
Prompt processing (PP) throughput increases from 300 t/s to 1000 t/s with the general kernel, and reaches 1400 t/s with the fused kernel.
These optimizations address a severe performance drop (from ~1500 t/s to 300 t/s) caused by applying small 35 MB LoRA adapters to models like Qwen3.8-Flash-Next.
The code is currently optimized for Strix Halo and is available in a fork, with plans to upstream to mainline llama.cpp or strix-llama.cpp pending architectural changes.

lora llama.cpp inference-optimization strix-halo qwen3.8-flash-next

Read originalShare on Threads

This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.

Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.