Teaching AI Agents to Ask Better Questions Using the 'Battleship' Game

Researchers at MIT CSAIL and Harvard SEAS built a new testbed using the Battleship game to help AI agents efficiently seek information in uncertain environments.

Measuring AI's question-asking ability through the BattleshipQA dataset, they found that small models struggle to generate useful questions. To address this, the researchers introduced two key strategies.

• Monte Carlo inference strategy: This approach treats the model's possible guesses as individual particles, weighting plausible guesses based on the observer's answers. Applying this, Llama 4 Scout's win rate against humans surged from 8% to 82%, surpassing GPT-5's performance, at a cost of only about 1%.
• Python code conversion (auto-formalization): This method converts natural language questions into clear encoded instructions, prompting the observer model to verify the answer. This approach significantly improved model accuracy, boosting GPT-4o-mini's performance by about 30%.

This research is expected to hold great potential for needle-in-a-haystack domains, such as scientific discovery, where AI agents must find the correct answer among a vast number of options.