Pinterest Introduces Loop Engineering to Improve Analytics Agent Compliance

Details

Pinterest developed a "loop engineering" framework to improve the compliance of its Analytics Agent, which answers data questions by following specific rules like inspecting metadata and searching all relevant sources.
The approach achieved 89.3% compliance across tuning and validation cases, though most of the improvement came from the first accepted change, with later iterations adding little value.
The system uses paired test suites (required and skip cases) to ensure the agent follows rules when needed and avoids unnecessary actions otherwise, preventing regressions where the agent might over-search or ignore valid shortcuts.
Each iteration starts with a fresh context and a structured handoff containing current scores, failing cases, and previous failures, reducing model inference costs by about fivefold compared to persistent session designs.
Changes are accepted only if they pass three safety checks: a repair/break gate, an acceptance critic to prevent hard-coding, and a regression guard to protect unrelated capabilities.
The process revealed that 10.3% of test cases changed verdict between identical runs due to model variation, necessitating evaluation of individual case changes rather than relying solely on aggregate scores.
The loop is limited to modifying the agent's system prompt, skills, context layer, and tools, with shrinking line limits to prevent unnecessary rewrites and ensure targeted fixes.

Key point