MoE Castration Emerges

A Qwen3.6-35B-A3B abliterated version has been released on Hugging Face.

The key point is that ablation of MoE models differs from dense models. It explains that refusal signals are more strongly tied to the expert path than to attention, so general Q/K/V LoRA alone doesn't work well.

The Abliterix method applied is as follows.

• Apply LoRA rank-1 only to O-proj and MLP down-proj
• Intentionally disable Q/K/V
• Project the refusal direction across all 256 expert down_proj slices in each layer
• Find the top 10 safety experts and lower the router bias to -2.10
• Apply orthogonalized steering vectors and Gaussian decay across layers
• Explore strength in the 0.5~6.0 range to avoid broken output

The evaluation result is 7/100 refusals, maintaining relatively close KL 0.0189 compared to the base model. The baseline is 100/100 refusals.

However, the author points out that the 0~3/100 figures on other abliterated model cards are usually underestimates from 30~50 token generation + keyword detection. They claim this evaluation is stricter, using Gemini 3 Flash judging, 150 token generation, and treating garbled output as refusal as well.