Composo Releases DishwasherBench: A Physics-Based Benchmark for LLM Agents

Composo Limited has released DishwasherBench (v0.1.0), an open-source benchmark and game named Spotless designed to evaluate LLM agents on physical reasoning and spatial planning. The system uses a rigid-body physics simulation of a real Bosch SMV4HAX40G dishwasher to score how effectively models can load dishes for the Eco 50 wash program.

Simulation and Scoring Mechanics

The benchmark relies on a deterministic physics engine (Jolt) to simulate loading, followed by a GPU-accelerated washing simulation that models jets, water droplets, and chemical reactions. Scoring is based on the number of items deemed "clean" after the wash, defined as having ≤ 4 specks and ≤ 4 mm² of visible soil. The simulation is calibrated against a single real-world wash record, though the developers note that results for stacked plates and cutlery are currently unreliable due to limitations in modeling overlapping items.

Model Performance and Limitations

Initial tests involved two models, gpt-6.1-sol and gpt-6-astra, playing three levels via OpenAI-compatible endpoints. The models generally outperformed the "careful" scripted baseline by placing more items, though they avoided critical errors like placing hollow vessels upside down. Key limitations include:

• No Human Baseline: There are no human player scores for comparison.
• Unverified Paths: The Anthropic API integration has not been tested live.
• Hardware Constraints: The simulation requires a GPU capable of running WebGPU (via Dawn) and has only been verified on macOS (Apple M3 Pro). Linux and Windows are untested.
• Resource Usage: Official tier washes take ~5 minutes on the reference machine and consume 1–2 GB of memory.

Availability and Integration

The project is licensed under MIT and available as a web app, an MCP server for agent integration, and a benchmark harness. It supports both "fast" and "official" simulation tiers, with the latter providing the authoritative scores. Composo emphasizes that this is a simplified model of a single machine and not a statement on actual product performance.