Nonobench: Open Benchmark Evaluates 49 LLMs on Nonogram Puzzle Solving

Nonobench is a new open-source benchmark measuring how well 49 LLMs solve nonogram (picross) puzzles. Models receive row and column clues and must return the full grid in a single attempt without using external tools.

Benchmark Structure

The evaluation consists of two modes:

• Standard mode: 30 puzzles ranging from 5x5 to 15x15 grids, sourced from the Nonograms dataset by Moyà-Alcover.
• Hard mode: Ten random 20x20 grids, each verified to have a single solution. Five of these cannot be solved by line logic alone, and random fills are used to prevent models from guessing picture-based patterns.

The benchmark tests 130 variants across different effort levels, executed via OpenRouter and pinned to each lab's specific model versions.

Key Results

Performance degrades significantly as puzzle complexity increases:

• Standard mode: Solve rates drop from 85% on 5x5 grids to 46% on 10x10 grids, and 20% on 15x15 grids, with each model evaluated at its best effort level.
• Key Performers: GPT-6 Astra solved all 30 Standard puzzles. In Hard mode, Claude Opus 5.5 solved 8 out of 10 puzzles, while 11 of 15 models failed to solve any.

Methodological Notes

To ensure accurate logic testing, Hard mode answers are provided as an array of 20 row strings rather than a single 400-character string, preventing models from losing track of the grid structure. The benchmark acknowledges noise in single-attempt results and provides 95% confidence intervals.

The project is open source under the MIT license, with code available on GitHub and results hosted at nonobench.com.