How Hyundai AutoEver Built an Incident Response Automation Agent for Its Big Data Cluster on Amazon Bedrock

Hyundai AutoEver's Data Platform Technology Team operates a Hadoop-based big data cluster 24/7, and to automate repetitive incident response, they built an agent combining Amazon Bedrock and LangGraph.

They viewed incident response as a 5-stage lifecycle of Detection, Investigation, Recovery, Post-Incident Review, and Improve, tying together alert confirmation, log exploration, RCA, recovery planning, post-incident reporting, and knowledge accumulation into a single flow. The goals were reducing MTTA, standardizing quality, and turning incident knowledge into an asset.

The architecture is divided into a VDI Agent and a Main Agent Server. On the VDI side, the Outlook Monitor detects incident emails and sends the incident payload, the Teams Agent handles progress updates and Q&A, and the SSH Runner executes read-only diagnostic commands. The server side consists of a FastAPI REST Gateway, LangGraph Orchestrator, PostgreSQL, Amazon OpenSearch Service, and Amazon Bedrock.

The workflow is designed with 14 agents, and currently 9 nodes covering classification/diagnosis through analysis are in operation. PostgresSaver checkpointing saves state at the node level, enabling resumption after interruption, and allows recovery, approval, and reporting nodes to be progressively expanded even while in operation.

The core design is a structure where nodes return only deltas instead of overwriting state, and a Reducer merges the parallel results. Fields like analysis, log, and timeline are merged using reducers such as _merge_dict and _add_list, and RCA-A and RCA-B each analyze independently before the Reflector cross-validates them.

The main nodes play the following roles:

• Coordinator: Classifies the service/component from the alert email, and verifies whether the cluster actually exists via the Meta DB to determine whether to proceed with the workflow.
• State Checker: Repeats Ambari snapshots and SSH diagnostics to objectively describe the current state. It can loop up to 50 times, and updates the alert status every 5 steps.
• Log Investigator: Prioritizes using OpenSearch aggregation, then generates exploration intent, and combines a low-cost model's Triage with a high-performance model's Summarize to organize evidence into an EvidencePack.
• Parallel RCA Analysis: Two models independently analyze the cause based on the same EvidencePack, each generating a Falsification and calculating a robustness_score.

Log exploration starts by default with a 10-minute window before and after the incident time, expanding to 20 minutes, up to a maximum of 30 minutes if needed, and log levels also expand in the order CRITICAL → WARN → INFO. Component names are normalized through synonym mapping, whitelisting, and fuzzy matching to reduce OpenSearch search failures.

The parallel RCA design is meant to reduce confirmation bias and hallucination from a single LLM. Different top-tier reasoning models review the same evidence and reach conclusions independently, after which the Reflector compares the argument structure and strength of falsification to make the final determination.