Reverse Engineering Apple ANE: LLM Inference Bottleneck Is Data Movement, Not Compute; Integrated into GPU in M5

Reverse engineering analysis of the Apple Neural Engine (ANE) confirms that the core bottleneck in LLM inference is not compute performance but data movement (dataflow). In particular, the integration of ANE cores into the GPU in the M5 chip in 2025 signals a shift in design philosophy for Transformer workloads, alongside a change in form as an independent hardware block.

ANE Architecture and Compute Structure

The M1 ANE consists of 16 parallel cores, with each core having 128 FP16 or 256 INT8 MAC lanes, forming a total of 2048 parallel MAC lanes. The hardware does not encode CNN layers and keeps partial sums local to avoid external memory access. Precision is FP16 input, Q16.16 fixed-point accumulation, and FP16 output. Nonlinear Activation is applied pointwise immediately after MAC with no intermediate memory round-trips. The ANE is a fixed-function dataflow engine without an ISA, controlled via pre-compiled task descriptors (TDs).

Why Data Movement Is the Bottleneck

The fundamental reason the ANE is optimized for CNNs is memory movement, not compute. It assumed predictable reuse patterns in 2017 CNN models, but the autoregressive decode of Transformers broke this assumption. The M1 ANE delivers 11 TOP/s performance, but system DRAM bandwidth is 68 GB/s, equivalent to only 0.5 OP/byte for FP16. The Roofline ridge point is 162 OP/byte, meaning arithmetic intensity must be at least 162:1 to avoid the DRAM bandwidth bottleneck. Below this ratio, increasing compute speed does not lead to improved token/s.

Memory Hierarchy and Execution Mechanism

The memory hierarchy consists of Unified DRAM, Shared ANE L2 (2 MiB), per-Core L1, and KMem. KMem is 64 KiB SRAM per core for Kernels, and L2 is 2 MiB shared across all cores. There are three types of DMA engines: KernelDMASrc (DRAM→KMem weights), TileDMASrc (DRAM→L2), and TileDMADst (L2→DRAM). Kernels are loaded from DRAM, not L2, and KernelDMA and TileDMA are processed serially, not in parallel, falling short of the DRAM ceiling (100 GB/s). Therefore, ANE decode performance is limited by DRAM bandwidth, and a 2.5x improvement (10→25 tok/s) requires approximately 2.5x bandwidth increase.