LLMs Are Not the Black Box You Were Promised

LLMs are no longer a 'black box' whose insides cannot be known. As Mechanistic Interpretability techniques advance, it is becoming possible to reverse-engineer and understand how neural networks work internally. In particular, Anthropic's 2025 research paper 'On the Biology of a Large Language Model' presents an important milestone in this field.

Solving the Superposition Problem Directly interpreting LLMs is very difficult due to the phenomenon of superposition, where a single neuron participates in multiple unrelated concepts, and a single concept is distributed across multiple neurons. To address this, researchers train a 'replacement model' to decompose the model's activations into sparse features that humans can understand. This makes it possible to identify high-level concepts such as 'Texas' or 'Olympics'.

Circuit Tracing Technique By tracing how the identified features interact with each other during the forward pass, a wiring diagram showing the computational process can be constructed. This allows us to grasp the logical flow inside the model.

The Reality of Multi-step Reasoning Through real examples, we can confirm that the model performs genuine multi-step reasoning via intermediate concepts. For example, for the question "What is the capital of the state containing Dallas?", the model operates in the following order.

• Dallas feature activates
• Texas feature activates
• Austin feature activates

This process shows that the model tracks semantic relationships between high-level concepts, performing a kind of pseudo-symbolic inference.