ARM Local Inference Speed

Measured llama.cpp token speeds on the ASUS Zenbook A16 with Snapdragon X2 Elite Extreme (X2E94100, Qualcomm Oryon Gen 3).

• 18-core CPU, 48GB unified memory, approximately 228GB/s bandwidth
• ISA: support for NEON, FMA, DOTPROD, I8MM, SVE/SVE2, SME/SME2, fp16, with a 4096-bit Matrix Engine (SME2) confirmed
• KleidiAI (SME2) appeared not to work on Windows, and the Adreno GPU was also tried but produced no output, so testing was done CPU-only

Performance was compared based on Q4_K_M.

• Qwen3-4B: 248 t/s (PP512), 42 t/s (TG128)
• Gemma-4-31B-it: 39 t/s (PP512), 6.5 t/s (TG128)
• Gemma-4-26B-A4B-it: 168 t/s (PP512), 31 t/s (TG128)
• Qwen3.6-35B-A3B: 171 t/s (PP512), 33 t/s (TG128)

For the same Qwen3.6-35B-A3B, changing quantization and runtime settings:

• Q4_K_M: 171 / 33.0 t/s
• Q5_K_M: 153 / 30.4 t/s
• Q5_K_M + q8_0 KV + FA(opencode): 145 / 29.6 t/s

The author rated Q5 Qwen3.6 as quite usable in opencode, and noted that local execution is possible even on battery. The next experiment goal is to run a Whisper model at 100% NPU.