Kakao Reveals Case Study on Automating KakaoTalk Recommendation Metrics Analysis on Hadoop Using AI Agents

Kakao's Social Recommendation Engine team revealed a case study on automating KakaoTalk recommendation metrics analysis on Hadoop using AI agents. The core approach was not to build a new analysis platform, but to organize the access, query, and interpretation procedures previously performed by humans into Agent Skills (Markdown documents) readable by agents, effectively layering them onto existing infrastructure like glue.

Skill-Based Automation Structure

Developers deployed repetitive procedures such as Hadoop client installation, Kerberos authentication, table exploration, and query writing as the hadoop-butler plugin. The AI reads these instructions, writes and submits SQL based on natural language requests, and derives insights from log data. This process utilized existing scripts with almost no new code written.

Context Documents and Error Prevention

To improve the AI's analysis accuracy, context documents such as CLAUDE.md or AGENTS.md were utilized. By specifying table definitions, feature meanings (e.g., watch_length, valid_view), and aggregation criteria, the AI was prevented from making mistakes such as confusing user_id with session_user_id or writing multiple COUNT(DISTINCT) queries that degrade performance in Hive.

Verification Loop and Regression Testing

To ensure the quality of the agent's output, a multi-layered verification pipeline using an LLM Judge was built. The agent was run in headless mode to obtain actual results, and PASS/FAIL judgments were made by comparing query execution traces with final outputs. This enabled the identification of regression issues that could arise from instruction modifications before deployment.