AI BriefingStudy Finds LLM Refusal Behavior Mediated by Single Direction
·2026.11.08 10:56
Key point
Researchers demonstrate that removing a single vector from the residual stream of 13 open-source models up to 72B parameters effectively disables refusal mechanisms.
Details
- Refusal behavior in large language models is mediated by a one-dimensional subspace, identified via difference-in-means on residual stream activations.
- Ablating this single direction bypasses safety filters, while adding it induces refusal on harmless instructions, proving it is a necessary and sufficient condition for refusal.
- A white-box jailbreak method using rank-one weight orthogonalization achieves high attack success rates on HarmBench without requiring gradient-based optimization or harmful completion data.
- The intervention preserves model coherence on MMLU, ARC, and GSM8K benchmarks, with performance changes remaining within the 99% confidence interval of baseline models for most architectures.
- Adversarial suffixes work by suppressing the propagation of the refusal direction and hijacking the attention of specific heads away from the harmful instruction content.
- The method highlights the brittleness of current safety fine-tuning, as a 70B parameter model can be jailbroken with less than $5 in compute using this weight edit.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
MIT CSAIL Research Demonstrates LLMs Can Natively Read and Write Neural Network WeightsStealthium demonstrates detection of ROME attacks via GPU memory weight tampering
Conversation Between Amazon CEO and US Officials Triggers Crackdown on Anthropic Models