Kakao Reveals AI Collaboration Evaluation Framework and AI-Based Hackathon System Case Study

Kakao has established an evaluation framework to continuously measure and improve the performance of AI models and human-AI collaboration processes. In quantitative evaluation, various metrics tailored to business characteristics are utilized, including precision, recall, F1 score, and response latency, in addition to accuracy. For code generation AI, metrics such as bug inclusion and coding convention compliance are added, while qualitative evaluation verifies usefulness through actual user interviews and feedback from internal experts.

In the '10K Hackathon' case study at Kakao AI Campus, an AI judging system for 250 participants was built in 48 hours. Project proposals were automatically generated with Gemini, UI was designed with v0, and backend and frontend code based on Spring AI and Kotlin were automatically generated using Claude Code. AI agents handled everything from API specification design to front-end and back-end integration, maximizing development speed.

However, during the deployment stage, a combination of session inconsistencies in the load balancer environment and Kubernetes configuration errors occurred, causing the AI to focus only on single issues and fail to find a solution. Ultimately, the problems were resolved through developer log analysis and code reviews, suggesting that human experience remains essential for diagnosing complex architectures and handling exceptions. Additionally, as the codebase grew and AI context management became difficult, the role of developers as architects rather than mere assistants was emphasized.