Fool's Gold (18-minute read)

The refusal mechanisms of open-weight LLMs can be removed in minutes via a lightweight weight editing technique called abliteration. While existing defense techniques failed to stop attacks, Fool's Gold adopts a defensive deception strategy that allows attacks but contaminates their results.

This technique includes attack simulations in the training loop, fine-tuning the model to generate confident fake (decoy) answers to harmful questions even when refusal mechanisms are removed. The fake answers possess the same tone and confidence as actual successful attacks, but their core elements are false.

Applied to 7 models across 5 families (9B–122B), the results showed that 51–90% of attackers switched to fake answers for harmful prompts that had not undergone defense training. This represents an improvement rate of +0.27 to +0.84 attributable to the defense technique itself, while general performance benchmark scores such as MMLU remained within the margin of error.

The security effect is epistemic. Attackers without an independent source of truth cannot distinguish between fake and actual answers. In the case of the 122B model, for CBRNE-related harmful questions, 82–86% of the defense model's answers were fatally incorrect, yet attackers could not detect this.