A 'diff' Tool for AI: Finding Behavioral Differences in New Models

AI model evaluation tends to miss unknown unknowns that existing benchmarks can't catch. To address this, the paper proposes cross-architecture model diffing, which automatically surfaces only the differences between two models—much like a code diff used for model updates.

The core is the Dedicated Feature Crosscoder (DFC). While existing crosscoders are good at capturing only shared concepts, DFC is designed with a separated structure—shared / English-only / French-only—so that behavioral traits unique to a single model aren't missed.

This method directly identifies behaviors that emerge in a new model, then uses steering to suppress or amplify that trait to verify whether it's actually causal. As a result, the following model-specific features were confirmed:

• Chinese Communist Party alignment in Qwen3-8B and DeepSeek-R1-0528-Qwen3-8B: suppressing it leads to more answers to Tiananmen-related questions, while amplifying it strengthens pro-government speech.
• American exceptionalism in Meta Llama-3.1-8B-Instruct: amplifying it reinforces narratives of American superiority.
• Copyright Refusal in OpenAI GPT-OSS-20B: suppressing it weakens copyright refusals, while amplifying it causes excessive refusals.

The authors do not make definitive claims about how these features arose. They could be the result of intentional training, or unintentional behaviors that emerged from the data.

In summary, DFC is not a silver-bullet tool that catches every risk, but it is a screening tool that filters, with high recall, differences unique to a new model out of thousands of candidate features. In particular, by comparing before and after model updates, it may be possible to catch newly emerged issues—like GPT-4o's sycophancy—more quickly.