A1Mobilsoft's Journey to Automating Operations Inquiries (2): Implementing Three Agents and Human vs. AI Verification

A simple vector search approach had the limitation of failing to filter out 'wrong answers' that were keyword-similar but contextually different. To solve this, we built a 3-stage process consisting of Retrieve → LLM Rerank → Diagnosis.

In Stage 1, we lowered the score threshold of Bedrock Knowledge Base to 0.4 to widen the search scope, and in Stage 2, the Haiku model classifies the inquiry type and re-evaluates the score. The final Stage 3, the Diagnosis stage, serves as the final gate that strictly determines whether the current inquiry and past cases truly match.

In particular, the Diagnosis stage was designed not to rely solely on the confidence score, but to include a separate boolean value called match, explicitly judging negative conditions such as 'when the purpose differs' or 'when the target/environment differs'. This achieved the result of reducing false positives to virtually zero.

All stages used the Claude Haiku model for cost efficiency and fast response speed. Backtest results confirmed that for domain-specific tasks, the difference in judgment accuracy compared to the Sonnet model was not significant.