Integrating AI into SRE Operations — Part 2: Automating Incident Root Cause Analysis with Alert Adviser

The SRE team at Yeogi-eottae Company developed Alert Adviser to enable QA, other service teams, and others beyond on-call personnel to quickly identify incident root causes. Previously, personnel had to manually check dashboards and logs after receiving alerts, but now AI traces primary observability data and provides verified reports.

Using Slack as the primary interface, it receives Grafana alerts in real time. Through MCP (Model Context Protocol), it directly queries observability data such as Mimir, Loki, and Tempo, and publishes analysis results separated into severity, estimated cause, and action guide. It runs on a single EC2 instance with a Go backend and a React dashboard, integrating with existing alert infrastructure without requiring separate infrastructure scaling.

To prevent AI misjudgments, three control mechanisms were applied.

• Evidence Grade Enforcement: Backend code uses regular expressions to verify measured values and speculative language, determining the evidence grade.
• Mutation Permission Blocking: System mutation commands like kubectl are removed from analysis responses and restricted to read-only.
• Misjudgment Pattern Verification Rules: Based on past error cases, it enforces hypothesis cross-validation and specific metric queries (e.g., checking I/O Wait during CPU alerts).

For internal document search (RAG), they abandoned self-hosting due to data cleansing burdens and search quality degradation, opting instead for MCP-based RAG tools from the company's common platform. This allows reflecting organizational knowledge while applying filtering logic to reduce the inclusion of irrelevant documents.