Garry Tan's "Skillify" — A Methodology for Turning AI Agent Failures into Permanent Structural Fixes

Y Combinator CEO Garry Tan proposes "Skillify", a methodology for quality control of AI agents. It starts from the problem that existing frameworks like LangChain provide testing tools but lack a workflow for "what to test and in what order."

Skillify refers to the process of converting an agent's failures into a permanent 'skill' that combines a markdown procedure, a deterministic script, and an automated test, instead of simply responding with a prompt fix.

The core design principles are as follows:

• Separating Latent vs. Deterministic: Clearly distinguishes between areas that require LLM reasoning (Latent) and precision areas that require code execution (Deterministic). It defines it as a major bug when an LLM is left to reason about tasks—such as timezone calculations—that can be immediately solved with code.
• 10-Step Verification Process: A skill is only recognized as such after passing strict steps including writing SKILL.md, unit tests (vitest), integration tests, LLM-as-judge evaluation, and E2E smoke tests.
• Workflow-Centric: If a framework is a 'gym membership,' Skillify corresponds to a specific 'workout routine.'

It emphasizes that agent skills, just like software code, decay without testing, and that as systems grow in scale, managing the discoverability of skills will become a key challenge.