Personal Experiments Show LLMs Can Directly Edit Weights to Solve Specific Tasks, But Struggle with Complex Benchmarks
·2026.11.17 23:48
Key point
A researcher conducted personal experiments using Claude Opus 5.5 for Weight-Space Programming, finding it excels at specific pattern-based tasks like Two Spirals and Brackets but underperforms gradient descent on complex benchmarks like MNIST and Shakespeare.
Details
- The author built a harness to test Weight-Space Programming (WSP), where an LLM edits neural network weights directly without gradient descent, using Claude Opus 5.5.
- The agent achieved strong results on specific tasks: 99.9% on Two Spirals, 100% on Bracket Prediction, and 86.2% on IMDB sentiment analysis (0.8 points below gradient descent).
- However, the agent underperformed on other benchmarks: it scored 64.1% on MNIST (compared to >85% for gradient descent) and only improved an undertrained Shakespeare LSTM from 39.9% to 40.5%.
- The experiments suggest WSP is effective for tasks where the LLM can explicitly reason about patterns (like geometry or simple rules) but is not a replacement for gradient descent on complex, data-heavy tasks where implicit learning is superior.
- The harness restricted the agent to a Docker container with no internet, limiting it to NumPy operations and a fixed number of evaluation turns to prevent cheating.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.