pi-autoresearch: Open Source That Generalizes Karpathy's "AI Autonomous Experimentation" Idea

pi-autoresearch is an Extension + Skill package that runs on top of the terminal AI coding agent pi, extending Karpathy's autoresearch concept beyond a specific model training task into a general-purpose optimization loop.

The core structure is "try → measure → keep if better, discard if not → repeat." pi's Extension adds tools, and Skill defines the order in which those tools should be used. This means it can be installed with a single line, pi install <github-url>, and attached to existing workflows.

Karpathy's original autoresearch was a minimal experiment released in March 2026, in which an agent modifies train.py, then after 5 minutes of training, keeps the change if validation loss improves and discards it otherwise. The idea was to repeat this overnight so you'd have logs and an improved model by morning, but it had limitations: it was NVIDIA GPU only and restricted to LLM training.

pi-autoresearch removes these constraints, making it possible to optimize anything measurable.

• Test speed: pnpm test
• Bundle size: pnpm build && du -sb dist
• LLM training: uv run train.py
• Lighthouse score: lighthouse --output=json

To allow long-running experiments to continue even if context is interrupted, session state is preserved in two files.

• autoresearch.jsonl: an append-only log of every experiment
• autoresearch.md: a summary of goals, attempts, blockers, and key achievements

Even if a new agent takes over, it can restore prior context just by reading these two files, effectively serving as external memory.

Also, after 3 or more experiments, it computes a confidence score based on MAD (Median Absolute Deviation) to distinguish whether an improvement is real or noise.

• 🟢 ≥ 2.0×: likely a genuine improvement
• 🟡 1.0–2.0×: better than noise, but marginal
• 🔴 < 1.0×: within noise range, rerun recommended

To ensure optimization doesn't break correctness, attaching autoresearch.checks.sh automatically runs tests, type checks, and linting after each benchmark. This is a system-level safeguard against situations where a quick performance gain breaks code quality.

Once experiments are done, the autoresearch-finalize skill groups the kept results into logical changesets and separates them into independent git branches. They're split so there are no file conflicts, allowing each branch to be reviewed and merged separately.

Cost control was also considered. To prevent the autonomous loop from continuously consuming tokens, it provides API key limits and maxIterations.

Ultimately, this project is an attempt to turn Karpathy's idea of "AI doing research on its own" into something usable in real-world development. The key point is that it aims to make real a workflow where you run it overnight and review the PR in the morning — including session restoration, statistical confidence judgment, correctness verification, and Git-native cleanup.