StarSkirmish Bench v0.2: GPT-6 Astra Leads LLM StarCraft Bot Coding, Human Bots Still Dominate

Benchmark Overview and Results

StarSkirmish Bench v0.2 evaluates how effectively LLMs can write StarCraft: Brood War bots in C++ within a 1-hour wall-clock limit. The benchmark tests long-horizon reasoning and agentic coding by requiring models to incorporate learnings from practice games. Each model runs in its respective lab's 1st party CLI harness (e.g., Claude Code, Codex, Antigravity CLI) at the highest available reasoning effort.

GPT-6 Astra secured the top score with a 57% average win rate against a roster of competitive human-written and demo bots. GPT-6.1 Sol followed with 46%, while Gemini 4 Argon, Claude Opus 5.5, and Claude Fable 5.1 were functionally tied in the 37–39% range. GPT-6.1 Sol was noted for offering the best value, scoring second at just over 1/10th the API cost of GPT-6 Astra.

Performance vs. Human Experts

Despite high scores against mixed rosters, all LLMs lag significantly behind top-tier human-written bots. Against Stardust, PurpleWave, and BananaBrain, the best model (GPT-6 Astra) won only 7% of games. GPT-6.1 Sol won 4%, while Gemini 4 Argon and Claude Opus 5.5 won 2%. This indicates that while LLMs can generate competent code, they struggle to match the strategic depth of specialized human AI agents in complex, real-time strategy environments.

Model Behaviors and Variance

• Gemini 4 Argon: Uniquely preferred one-shot coding, resulting in the fewest builds and code versions. However, it exhibited high run-to-run variance (18-point spread between best and worst runs) and won only 7% of games against high-tier opponents despite practicing against them 41% of the time.
• Muse Wonder Full: Focused on macro strategies and upgrades (2.5 per game), but practiced almost exclusively against easy opponents (1% against Tier B or harder), leading to a low score of 21%.
• Practice Correlation: The strongest predictor of a model's score was the amount of time spent practicing against the 7 strongest bots on the roster. Models like GPT-6 Astra and Gemini 4 Argon engaged with strong opponents early, while lower-scoring models avoided them.

Implementation and Scoring Changes

v0.2 replaced Elo ratings with plain win rates (0–100) to allow for direct comparison across model generations and parallel evaluation. The setup involves Protoss vs. Protoss matches on three maps (Heartbreak Ridge, Benzene, Destination) using BWAPI 4.4.0 and OpenBW. Context is compacted at approximately 250k tokens, and web access is disabled. StarSkirmish scores show low correlation with general intelligence indices like the Artificial Analysis Intelligence Index (ρ 0.26) but high correlation with specialized coding benchmarks like Vals IOI (ρ 0.95) and WeirdML v3 (ρ 1.00).