AI BriefingRubric Design and System Implementation for Skill Quality Management
Key point
Toss's AI DX team built a 30-item Rubric system combining rule-based and model-based approaches to manage the quality of Skills for coding agents.
Details
Skill, an auxiliary tool for coding agents, is an artifact called and read by the LLM, and unlike code, it lacks compilers or testing tools, making quality management difficult. In particular, trigger failures caused by poorly written descriptions and invocation failures caused by format violations emerged as major problems.
To address this, a Rubric composed of 6 sections and 30 items was designed. The core design principle is to separate management so that deterministic defects are handled via Rule-based checks, while semantic judgments are handled via Model-based checks.
Main Structure and Operation of the Rubric:
- 6 sections: Validity, Structure, Trigger, Content, Resources, Safety
- Severity classification: Managed by dividing into BLOCKER, MAJOR, and MINOR
- Grading system: Divided into 5 levels from S to F, and if there is even one BLOCKER item, an F grade is immediately assigned to block deployment
This separated design is also effective for optimizing operational costs. By first performing rule checks, which incur almost no cost, and proceeding to the costly LLM model checks only for items that pass, efficiency is increased.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Toss Doctor V2 Released: Maximizing QA Efficiency with LLM-based Automated Testing and Self-healingToss Bank Introduces 'Stylepack' Feedback Loop to Resolve Coding Agent Team Rule Violations