LG AI Research Proposes ATAD Dynamic Protocol to Address Static Benchmark Limitations at ICLR 2026

LG AI Research introduced ATAD (Agent-Centric Text Anomaly Detection) at ICLR 2026, a dynamic evaluation protocol designed to overcome the saturation and data contamination issues of static benchmarks like MMLU and GSM8K.
Experiments with GPT-4o, Claude-3.5-Sonnet, and Gemini-2.0-Flash showed an overall performance decline of approximately 21.3-33.9% as the protocol iteratively increased problem difficulty beyond the models' base capabilities.
The Orchestrator agent significantly improved problem quality, raising Validity from 4.30 to 4.85, Coherence from 3.71 to 4.74, and Fairness from 3.20 to 4.65, while increasing the Approval Rate from 38.14% to 87.14%.
ATAD defines evaluation as a co-evolving protocol rather than a fixed dataset, allowing the benchmark to adapt to new models without rebuilding from scratch, and demonstrated stability in results across repeated generations.

Protocol Structure

Three agents interact: the Teacher generates problems, the Student solves them, and the Orchestrator validates clarity, logical consistency, and fairness.
Failure-Driven Sample Finalization confirms a problem as a benchmark item only when the Student fails, ensuring the difficulty matches the model's current limits.
The protocol includes seven specific Text Anomaly Detection tasks (T1-T7) to test cross-sentence logical reasoning and prevent superficial pattern matching.

No single model consistently achieved the highest performance across all seven tasks, revealing unique reasoning blind spots for each model.
Without the Orchestrator, GPT-4o accuracy was 68.29%, lower than the 72.43% with the Orchestrator, indicating that lower accuracy alone does not indicate a better benchmark if problems are ambiguous.
The Orchestrator prevents artificial difficulty caused by logical flaws or ambiguity, ensuring that performance drops reflect actual reasoning limitations rather than poor problem design.