AI BriefingArm KleidiAI Now Built into ExecuTorch 0.7
Key point
Arm KleidiAI is now enabled by default in ExecuTorch 0.7, greatly improving LLM inference performance on mobile and edge devices.
Details
Arm's AI acceleration layer, KleidiAI, is now enabled by default in ExecuTorch 0.7 beta, providing automatic acceleration not only on the latest Arm CPU architectures but also on existing mobile devices.
Key technical features are as follows:
- Use of SDOT (Signed Dot Product): Through the SDOT instruction introduced starting with the Armv8.2 architecture, matrix multiplication operations based on Int8 and Int4 precision are efficiently accelerated.
- Broad hardware support: SDOT is supported on approximately 3 billion Arm-based devices (about 72% of all devices), enabling GenAI execution not only on the latest flagships but also on devices released years ago and on the Raspberry Pi 5.
- Performance improvement: When running the Llama 3.2 model on the Galaxy S24+, it shows a 20%+ performance improvement in the prefill stage compared to without KleidiAI, achieving 350+ tokens/s during prefill and 40+ tokens/s during decode.
With this update, Android and cross-platform developers can achieve faster model startup, lower latency, and reduced memory footprint without any separate custom tuning.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Underdog Releases 'Husky', a Model-Specific Inference Engine Up to 4.5x Faster than MLX