QUBA Development Log: Using an AI Agent to Analyze Jira Bug Ticket Root Causes and Automatically Suggest Fixes

Sharing the development process of QUBA, a side project where an AI agent investigates Jira bug tickets registered by QA to propose root causes and fixes
Actual measurements of 7 QA tickets in August showed that using QUBA reports reduced tool calls by 12% (274 -> 240), cut costs by 6%, and took an average of 14.1 seconds for the initial investigation
Integrated a LangGraph-based Supervisor Agent with MCP Tools to access internal data such as Jira, code, and DB, and applied a Human-in-the-Loop approval gate
Embedding cosine similarity could not distinguish between correct and incorrect answers (0.75~0.82), but Cross-encoder reranker scores showed a 16~23x difference, enabling precise judgment

Architecture and Pipeline

Compared n8n, LangGraph, and Claude Agent SDK, and adopted LangGraph for explicit flow control and testability
Composed of 7 nodes from fetch_ticket to post_comment, where the LLM extracts 3 core keywords to generate search queries
Assembled a prompt within 4,000 characters, after which the LLM generates a JSON report including root cause, fix direction, and Fix Scope

Used BGE-M3 family 1024-dimensional vectors and chunking based on semantic boundaries per document type, achieving a 100% indexing success rate (all 346 documents)
Preserved unique identifiers (error codes, endpoints, etc.) as metadata rather than vectors to apply exact match filtering
Merged search results with different score scales using RRF (Reciprocal Rank Fusion) and mapped them to a 0~1 confidence range via log-scale recalibration