When agents research before coding, optimizations get better

Reading code alone to find optimizations had its limits. In llama.cpp's CPU inference path, fine-grained SIMD tuning barely moved performance, and the key issue was that text generation is a memory-bandwidth bound problem of reading model weights from DRAM.

So a research phase was added in front of the agent. Having it first read papers, competing forks, and other backend implementations before running experiments led it to start finding better hypotheses faster. In particular, looking at actual implementations like ik_llama.cpp, CUDA, Metal, and llamafile turned out to be more useful than arxiv.

Using 4 cloud VMs and about 3 hours, at a total cost of roughly $29, 30+ experiments were run, of which 5 made it into the final code. The key was not making the computation itself faster, but reducing unnecessary memory passes and fusing operations that had previously been separate.

The optimizations that were adopted are as follows.

• Softmax fusion: merged the 3 passes of copy, scale, and mask add into 1 pass
• RMS norm fusion: combined memcpy followed by scale into a single loop
• Adaptive from_float parallelization: dynamically branches on a row/element basis depending on prompt processing versus text generation
• Graph-level RMS_NORM + MUL fusion: found a pattern missing from the CPU backend and implemented it as a fused AVX2/NEON kernel
• Flash attention KQ fusion: merged scale, pad, mask, and max into an AVX2 FMA pass

The point where research mattered most was the fact that RMS_NORM + MUL fusion, which already existed in the CUDA and Metal backends, was missing from CPU. Looking only at the code, the two-step process seemed natural, but comparing it against other backends revealed the bottleneck, and that difference led to an actual optimization.

In the end, with flash attention turned on, faster results were obtained on both x86 and ARM. Overall, text generation improved +15% on x86 and +5% on ARM, confirming that an agent that researches first finds better optimizations than an agent that only reads code.