AI BriefingMeta-Harness: End-to-End Optimization of Model Harnesses
Key point
Meta-Harness is a system that automatically searches for and optimizes model harness code for LLMs through an agentic proposer.
Details
The performance of LLM systems depends heavily not only on model weights but also on the Harness—the code that stores, retrieves, and presents information. However, harnesses today are mostly hand-designed, and existing text optimization methods have the limitation of being ill-suited to this setting because they overly compress feedback.
Meta-Harness is an outer-loop system that directly searches for harness code for LLM applications. This system leverages an agentic proposer that can reference the source code, scores, and execution traces of all previous candidates through the file system.
Key experimental results are as follows:
- Online text classification: Improved performance by 7.7 points over existing state-of-the-art context management systems, while reducing context token usage by 4x.
- RAG-based math reasoning: Improved the average accuracy of 5 models by 4.7 points on 200 IMO-level problems.
- Agentic coding: Surpassed existing sophisticated hand-engineered baselines on TerminalBench-2.
These results demonstrate that rich access to past experience enables automated harness engineering.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.