Can SAEs Capture the Geometric Structure of Neural Networks

Neural network representations are rich with curved geometric structures, but straight-line tools are far easier to work with. Sparse Autoencoders (SAEs) are a representative interpretability tool that maps the internal world of neural networks using multiple directions in activation space.

However, individual SAE features fail to fully capture curved manifolds and only reveal partial glimpses. Research shows that the way SAEs capture geometric structure can be divided into three categories.

• Shattering: Tiling the curved structure by representing every point on the manifold as an individual, separate feature.
• Compact capture: Representing the manifold like a coordinate system using a small number of shared features.
• Dilution: Features are partially shared and each handles a specific region of the manifold.

In real neural networks, the Dilution phenomenon is mainly observed. Individual SAE directions capture meaningful local regions of the manifold, but the overall structure is distributed across multiple directions, making it difficult to understand the whole from individual features alone.

To address this, instead of interpreting individual features separately, we developed an unsupervised pipeline that groups features together to reconstruct the overall structure. This pipeline successfully found very rich and specific neural network geometric structures in Llama 3.1 8B.