CODA Kernel for Transformer Optimization Released

Transformer training systems are built around dense linear algebra, but memory-bound operations such as Normalization, Activation, Residual update have become the bottleneck for overall performance. This is because these operations repeatedly move large intermediate tensors to global memory relative to their arithmetic operation count.

CODA is a new GPU kernel abstraction technique that expresses these operations as GEMM-plus-epilogue programs. This approach is designed to perform related operations while the GEMM output tile remains on-chip, before it is written to memory.

The key features of CODA are as follows:

• Efficient data reuse: It fixes the GEMM main loop and provides composable epilogue primitives for scaling, reduction, and pairwise transformations.
• High performance: It achieves high performance not only with expert-written kernels but also with LLM-written kernels, demonstrating that framework-level productivity and hardware-level efficiency can be secured at the same time.