Taste-Bench: Frontier LLMs Achieve 59.7% Accuracy on Decision Forking Tasks

Taste-Bench evaluates the taste of LLM agents, defined as their ability to choose the better direction at a real decision fork in long-horizon tasks. The benchmark presents the model with a task, the trajectory up to the fork, and two candidate next steps, requiring it to select the step that the hidden rest of the trajectory proves correct. A wrong choice often appears reasonable initially but consumes the agent's budget later.

Benchmark Composition and Methodology

The dataset contains 502 questions derived from software engineering and machine-learning research trajectories, eliminating the need for expert annotation. Questions are constructed using two methods:

• Parallel forks: Independent attempts at the same task diverge at a specific point, with the final outcome labeling the better direction.
• Detour forks: An agent abandons a direction after failure and recovers later; the abandoned path and the recovery serve as candidates.

Of 4,657 mined forks, 502 survived filtering for triviality and undecidability. The set includes 390 engineering questions (from SWE-bench and SWE-bench Pro) and 112 research questions (from METR's MALT release).

Evaluation Protocol and Results

Models are evaluated using the paired_order_v1 protocol, where each question is asked in both the published option order and its reverse. A correct answer requires success in both orders, reducing random guessing scores to 25%. The best-performing model, GPT-5.6 Sol, achieved an average accuracy of 59.7%, followed closely by GPT-5.5 at 59.5%.

Other notable scores include:

• Claude Opus 5: 55.5% average
• Grok 4.5: 54.6% average
• GLM-5.2: 53.9% average

The benchmark highlights that even frontier models struggle with long-horizon judgment, as the best accuracy remains below 60%. The dataset is gated on Hugging Face to prevent training contamination, and the code is released under the MIT License.