AI Briefingllama.cpp, Hexagon NPU real-world benchmark
Key point
On the OnePlus 12's Snapdragon Hexagon NPU, llama.cpp achieved 4.5–12.5 tok/s depending on the model.
Details
On the OnePlus 12 (Snapdragon 8 Gen 3), llama.cpp's Hexagon backend was cross-compiled on Ubuntu and then run via Termux.
gemma-3-12b-it-qat-Q4_0: pp(prompt processing) 8 tok/s, tg(token generation) 4.5 tok/sgemma-3-4b-it-qat-Q4_0: pp(prompt processing) 20 tok/s, tg(token generation) 12.5 tok/s
The speed was similar to the Snapdragon 8 Gen 3 CPU, and there was almost no heat generation.
Supported GGUF quantization types are limited to Q4_0, IQ4_NL, MXFP4, Q8_0, and F32.
- KV cache quantization is not yet supported.
- If the model and KV cache exceed 4GB, multiple NPUs must be attached using
GGML_HEXAGON_NDEV=2and--device HTP0,HTP1.
The example run set LD_LIBRARY_PATH=./lib:/vendor/lib64, ADSP_LIBRARY_PATH=./lib, and GGML_HEXAGON_NDEV=2.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Why LLM Research Agents Generalize Without Benchmark Overfitting: 'Compressibility' Is Key
ArtificialAnalysis's LLM Intelligence-Cost Graph Misleads with Log Scale and Data Center Pricing