BrokenML Benchmark Evaluates AI Agents' Ability to Fix Distributed ML Workloads

The BrokenML benchmark evaluates whether AI agents can successfully repair broken machine learning workloads, specifically targeting PyTorch over Gloo on a two-node Slurm cluster and a llama.cpp server running in Docker Compose. A separate verifier re-runs each job on a fresh cluster to objectively score the solutions provided by the agents.

Fault Scenarios and Access Levels

The benchmark includes faults designed to reflect real-world operational issues, such as:

• Shape mismatches across ranks or hanging collectives
• Straggler nodes and OOM errors
• Broken model checkpoints and scheduling issues
• Poor serving throughput and silent loss divergence

Nine models were tested at three distinct access levels:

1. Config and code only
2. Config, code, and live read-only logs
3. Root access on the cluster

Key Findings and Limitations

The primary finding is that the performance gap between open and closed models is most pronounced when agents are restricted to config files and code alone, without full cluster access. This suggests that diagnostic capability without direct system control is a key differentiator. The study notes two main caveats: it uses CPU over Gloo (not GPUs) and performs only one run per task per model.