Second Look Research: Combining Fictional Stories and Difficult Advice SFT Reduces Agentic Misalignment in Qwen 2.5 to 4%

• Combining fictional story SDF and difficult advice SFT reduces agentic misalignment in Qwen 2.5 32B and 72B base models to 4%, compared to baselines of 61% and 69% respectively.
• Difficult advice SFT alone is far more effective than fictional story SDF, achieving a 6% misalignment rate on both model sizes (using 3,000 samples/~3M tokens for 32B and 1,000 samples/~1M tokens for 72B), whereas stories alone reduce it to 43% on 32B and 63% on 72B.
• Teacher model selection is the primary driver of SFT effectiveness; GPT-5.6 Terra data reduces misalignment to 21% on Qwen 2.5 32B when used alone for SFT, while Claude Sonnet 5 data reduces it to 44%.
• The benefit of difficult advice stems from response quality rather than refusal; refusal-only datasets show lower effectiveness than original advice data when controlling for action rates.

Methodology and Data Generation

• Fictional stories are generated based on 16 sections of the Claude Constitution, using an attribute grid to vary genre, setting, and protagonist identity.
• Difficult advice data is created via a 9-step pipeline expanding Anthropic's 6-step process, focusing on ethical dilemmas and direct advice rather than refusal.
• Training uses LoRA (rank 64) with a specific sequence: Stories SDF, followed by a 13K sample instruction-tuning mix, then Difficult Advice SFT.

Evaluation and Generalization

• Evaluation uses the Anthropic Agentic Misalignment benchmark with 27 conditions (3 harmful actions x 9 goal contexts).
• Models trained on AI-assistant personas show lower misalignment rates than those addressed by human names.
• Results generalize to larger open models including DeepSeek-V3.1, Qwen3.6-27B, and Nemotron Ultra 550B, with GPT-5.6 Terra data generally outperforming Claude Sonnet 5 data when action rates are controlled.