AI Experts and Auto-Hunting: AI Pipelines Controlled by a Harness

Musinsa's Logistics Platform Team automated everything from slow query analysis to PR creation and review reflection with query-engineer. It collects slow SQL via CloudWatch Logs Insights, traces the cause by following code paths and DB metadata, then generates a fix.

The conclusion from this project was that it's the harness, not the model, that determines agent performance. Asking a general-purpose AI without context tends to produce only textbook answers, so the team fixed a Spring Boot·JPA/Hibernate expert role via the system prompt and restricted output to unified diff patches. Since patches frequently broke, they added a normalizer to fix the order of deletion/addition lines, along with a 4-stage fallback strategy.

The pipeline works as follows.

• Stage 1: Collect slow queries via CloudWatch Logs Insights, and merge queries that differ only in literals into one using SQL fingerprinting.
• Stage 2~3: After gathering DB metadata, up to 5 specialists analyze it in parallel.
• Intermediate verification: After normalizing the patch, check syntax, result shape, EXPLAIN, and actual execution wrapped in LIMIT to verify consistency.
• Pre/post-PR loop: Before PR creation, an AI review loop fixes issues until CRITICAL/WARNING counts reach 0, and after the PR, review comments are re-analyzed and reflected back into the branch.

The review system runs up to 8 perspectives of review by using two providers simultaneously — Claude + OpenAI Codex. Results are deduplicated based on file + category + line, and CodexCliNotFoundError only fails the Codex perspective while the Claude perspective continues. On the other hand, SkillFileLoadError is handled separately to prevent gate malfunction. Overall progress is streamed simultaneously to the web UI and CLI qe via PipelineCallbacks and SSE.