An Engineer's Guide to Better AI Skills: Implementing a Test Process for Agent Optimization

After introducing a custom skill for the rx-mvvm architecture, Pinterest engineers found that skill invocation was inconsistent in Pin-agent (an internal fork of OpenAI Codex) and Claude Code. Invocation was especially shaky with short, ambiguous prompts, so they built a Bash-based test harness to quantify this.

The harness ran claude --print --verbose --output-format stream-json and parsed the logs to determine whether the skill was invoked. Using 15 positive prompts and 5 negative prompts repeated 5 times, they ran a total of 100 tests (5 runs × 20 prompts) and calculated core success rate, edge false positive rate, and overall accuracy. GPT 5.2-codex and Opus 4.5 were used for the comparison experiments.

The initial baseline showed overall accuracy of only 73% for Codex and 62% for Claude. They then raised invocation rates by adding more context (such as architectural components) to the skill frontmatter, adding strong directives like "YOU MUST LOAD THIS SKILL IF," and organizing a skills table with selection rationale in AGENTS.md.

• Applying multiple methods together produced the largest improvement, but the effect was more pronounced for Codex.
• In Claude Code, the same combination did not lead to the same gains.
• When the agent was asked to improve the additional content itself, the invocation rate actually dropped slightly.

In conclusion, skill loading is not something to leave to chance—it must actually be measured and tuned—and the most reliable approach is to combine clear intent, sufficient context, and specific instructions. All test cases succeeded when 'load this skill' was explicitly stated, and for domain-specific tasks, both prompt quality and skill design need to be improved together.