IBM Research ALTK-Evolve tops tau2-bench retail leaderboard with learned guidelines on GPT-4.1

·2026.10.21 00:19

Key point

The method uses GPT-4.1 to extract behavioral guidelines from agent trajectories, improving reliability and accuracy without weight updates or human annotation.

1 / 4

Details

IBM Research's ALTK-Evolve achieves 67.5% Pass^4 on the tau2-bench retail domain using GPT-4.1, outperforming all frontier models and the fine-tuned RAFT-30B-A3B (61.4%) by 6.1 points.
The system extracts prompt-level guidelines from agent trajectories without weight updates or human annotation, adding them to the system prompt as a markdown list.
On the airline domain, the GPT-4.1 agent with sup3 guidelines reaches 44% Pass^4 on the leaderboard, while the GPT-5.5 agent with lossless guidelines achieves 82.0% Pass^4, ranking first.
Adding guidelines increases input token costs by 32-65% but reduces conversation turns by 0.3-0.4, improving time-to-completion and allowing most cost recovery via prompt caching.

prompt-engineering gpt-5.5 agent-memory gpt-4.1 ibm-research altk-evolve tau2-bench

Read originalShare on Threads

This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.

Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.