Part 2: Policy-Following Evaluators, LLM-as-a-Judge

This summarizes how to build a human-expert-level LLM-as-a-Judge while turning Tinder AI-enabled Discovery's description generation policy into an actual evaluation system.

The evaluation method was set as Pointwise evaluation, the scale as Binary evaluation, and the criteria as Single-aspect. In a situation where a single model was being iteratively improved, dividing outcomes into pass/fail based on absolute criteria was the easiest to manage, and it also allowed the same structure to be maintained with human evaluators.

Initially, the Judge was built using only a generic prompt that fed in the policy document and few-shot examples as-is, but agreement with human evaluation reached only about 60%. The reasons were that the policy was a document written for generation, that the LLM could not replicate humans' implicit judgment, and that key conditions were easy to miss within the long policy document.

The problem was solved by decomposing the prompt in stages.

• Policy decomposition: The long policy was broken down into a detailed checklist, turning each item into an individual binary judgment.
• Critique decomposition: Rather than transcribing expert intuition as-is, critiques were restructured into a form the LLM could interpret, aligned with the checklist criteria.
• Adding a reminder: The key criteria were re-emphasized at the end of the prompt to reduce omissions.

As a result of repeating this process, 84% agreement was ultimately achieved. Considering that agreement among internal human evaluators was in the high 80s to low 90s percent, this reached a level usable in practice.

The LLM Judge was used not only for automated evaluation but also for policy improvement and debugging. Looking at the critiques in failure logs made it possible to quickly classify causes such as vague expression, incorrect information, and weakened connections, which directly led to fixes in the generation model or supplementation of the policy.