AI BriefingPick
Training 1,000 Weights in Gemma 3 270M Unlocks Hidden Sorting and JSON Skills
·2026.11.02 10:23
Key point
A study on Gemma 3 270M shows that training just 1,000 random weights (0.001% of the model) enables hidden skills like sorting and JSON output that prompts alone could not unlock.
Details
- Training 1,000 randomly selected weights in Gemma 3 270M raised sorting accuracy from 1% to 97% and enabled clean JSON output with only 30 weights, proving skills exist in the model that no prompt can elicit.
- The method, based on a NeurIPS 2025 study, uses sparse weight updates on a single 8 GB consumer GPU; 10 to 100 weights closed up to 50% of the performance gap to full fine-tuning on Llama and Qwen models.
- Multiplication remained weak, reaching only 33% accuracy with 100,000 trained weights, indicating the skill was largely absent from the base model, while addition improved significantly with 1,000 weights.
- Attention output matrices were the most effective location for weight selection, with 100 weights trained there achieving 47% sorting accuracy compared to 35% when picked randomly from the entire model.
- The skill relies on many small changes acting together; training only the 50 most important positions from scratch achieved 24% accuracy, outperforming the 16% achieved by training 50 random positions, while removing half of the 1,000 trained weights dropped accuracy to 25% or 15%.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
MIT CSAIL Research Demonstrates LLMs Can Natively Read and Write Neural Network Weights