AI BriefingGoodfire Research Finds Neurons Are an Interpretable Basis for LLMs
·2026.11.10 09:00
Key point
Analysis of Qwen3.6-35B-A3B shows that ablating 92.2% of weak neuron activations causes minimal performance loss, proving neurons are tractable units of analysis.
Details
- Neurons in MLP middle layers are a more appropriate basis for interpretability than distributed directions or learned feature spaces, as strong activations are essential to computation while weak ones are dispensable.
- Ablating the weakest 92.2% of MoE neuron activations (keeping only the top 320 per layer) increases language model loss by only 2.9% and causes minor benchmark fluctuations, such as -1.2pp on MMLU and -1.7pp on GSM8K.
- Most neurons operate in a single computational mode, with one mode accounting for over 90% of causal influence; natural language descriptions of these modes allow 95% accuracy in predicting neuron firing for strong activations.
- The study identifies specific subcircuits, such as one predicting the token "dance floor" in social contexts and another detecting camelCase identifier subwords in code, demonstrating that neurons perform distinct, interpretable computational functions.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Goodfire argues alignment is solvable through interpretability and intentional designGoodfire Unveils Activation Probe Technology to Detect Reward Hacking via Internal Model Signals
Study Finds LLM Refusal Behavior Mediated by Single Direction
MIT CSAIL Research Demonstrates LLMs Can Natively Read and Write Neural Network Weights