Anthropic Reveals Mathematical Framework Showing Induction Heads Are Key to In-context Learning in 2-Layer Transformers

An Anthropic research team led by Chris Olah presented a mathematical framework for interpreting the internal workings of Transformer models. The study analyzed Attention-only toy models to identify qualitative changes in algorithms across model layers.

Algorithm Differences by Layer and Induction Heads

• Characteristics of 1-Layer Models: A 1-layer Attention-only Transformer operates as an ensemble of Bigram and 'Skip-trigram' (A... B C pattern) models, allowing direct access to tables from the weights. This implements a very simple form of In-context Learning.
• Evolution in 2-Layer Models: In 2-layer models, much more complex algorithms are implemented through Attention Head Composition. Specifically, a special Attention Head called an 'Induction Head' forms, acting as a very general In-context Learning algorithm.
• Key Finding: 1-layer and 2-layer models use very different algorithms to perform In-context Learning. The Induction Head in 2-layer models represents a more sophisticated algorithm at inference time, marking an important transition point relevant to larger models.

Interpretive Approach

• Weight-Based Analysis: The research team revealed that the composition algorithms in 2-layer models can be detected directly from the weights. Additionally, they noted that neurons in MLP layers are generally Polysemantic, making them difficult to interpret, and that Superposition theory led to separate follow-up research.