Qwen-Scope: Decoding Intelligence and Unlocking Potential

Qwen has released Qwen-Scope, an interpretability tool that inserts and trains Sparse Autoencoders (SAE) into models of the Qwen3 and Qwen3.5 families. SAEs decompose the dense representations of hidden layers into sparse, disentangled features, enabling better understanding and control of a model's internal behavior.

The released weights consist of 14 sets of SAEs across 7 LLMs, including both dense and MoE models. The widths are 32K, 64K, 80K, and 128K, with L0 values of 50/100, and they were trained by sampling 0.5B tokens from each corresponding model's pretraining data. This configuration aims to achieve broad feature coverage, semantically distinct features, and stable training all at once.

The scope of application spans four areas.

• Inference: Without natural-language instructions, adjusting feature activations steers outputs such as language, entities, and style as intended.
• Data: Using only a small amount of seed data, features can be selected to perform classification, and inactive features can be leveraged to synthesize rare samples, improving long-tail coverage. Qwen states that efficiency improves by about 15x compared to existing methods.
• Training: By identifying abnormal activations such as code-switching or repetitive generation, the loss function can be adjusted during SFT, and during RL, sampling can be improved to better capture rare failure cases.
• Evaluation: By comparing feature coverage between test sets and benchmarks, redundant evaluations and coverage gaps can be identified.

Qwen emphasized that Qwen-Scope can be more than just an interpretability tool—it can serve as a model development tool that improves inference, data, training, and evaluation as a whole. A demo is available to try on Hugging Face and ModelScope.