Lessons Parameter Golf Left for AI-Assisted Research

OpenAI ran a constrained ML challenge called Parameter Golf, which required minimizing held-out loss on the fixed FineWeb dataset while keeping model weights and training code combined under 16MB, with training completed within 10 minutes on 8×H100. Participants received a baseline, dataset, and evaluation script and submitted via GitHub PR, and OpenAI ran the challenge for 8 weeks.

1,000+ participants made 2,000+ submissions, and for the record track, each submission was directly reproduced to confirm whether it was actually a record at the time of submission. Notable results included optimization that combined existing improvements well, quantization such as GPTQ-lite and full Hessian GPTQ, score-first per-document LoRA test-time training, self-generated GPTQ calibration, and new modeling/data ideas such as the CaseOps tokenizer, XSA, SmearGate·BigramHash, and mini depth recurrence.

The biggest change was the widespread use of AI coding agents. Agents lowered experimentation costs and reduced barriers to entry, but they also created new burdens for judging, attribution, and scoring. To handle hundreds of submissions coming in per day, OpenAI ran a Codex-based triage bot, and also dealt with the problem of strong ideas outside the rules being repeatedly replicated by other agents.

The non-record track was more experimental, and OpenAI picked 15 submissions to showcase technically interesting approaches. Half of them beat the naive baseline of 1.22 BPB, with the best result reaching 1.12 BPB, and RunPod's $1,000,000 compute sponsorship also enabled greater participation. OpenAI concluded that such open challenges can serve as a talent-discovery channel that reveals machine learning intuition and persistence.