LLM Game Navigation Benchmark Released

Although LLMs excel at complex mathematical reasoning, the benchmark DungeonBench has been released to evaluate how well they perform in simple dungeon game navigation that a child could solve.

Evaluation Method and Criteria

• Task: Navigate a 10x10 grid map, avoiding walls and achieving goals in the order of 'collect weapon > defeat monster > move to exit'.
• Failure Condition: A run is considered failed if there are 3 illegal moves.
• Settings: All models were tested with reasoning capabilities enabled.

Key Model Performance Results

• DeepSeek-V4-Pro (high): Ranked first with a perfect score of 12/12 and 0 illegal moves. It formulated efficient and accurate plans on all maps.
• Gemma-4-31B-it, Qwen-3.8-27B, GLM-5.3-Flash: Ranked in the top tier with a score of 11/12. Notably, Qwen showed persistence by eventually solving the problems despite requiring significant reasoning time.
• Muse-Glimmer-30B, DeepSeek-V4-Flash: Recorded a score of 10/12.
• Nemotron-3.5-Lightning-30B: Ranked at the bottom with a score of 5/12, committing 7 illegal moves.

This benchmark highlights the limitations of LLMs' spatial reasoning and planning capabilities. The code is available on GitHub, allowing anyone to test it directly.