Kakao Open-Sources 'S2Graph', a Large-Scale Distributed Graph Database

Kakao has open-sourced S2Graph, a large-scale distributed graph database applied to over 20 services including KakaoTalk and Daum. The system consists of a graph API server written in Scala and Play Framework, built on a modern tech stack including HBase, Kafka, and Spark.

Technical Challenges and Solutions

Kakao faced physical limitations due to increasing data scale in its existing manually sharded MySQL environment. S2Graph manages 200 million users and 10 billion relationships, processing 50 million relationship changes and 3 billion activity additions daily. It is designed to achieve real-time breadth-first search (BFS) performance, handling 65,000 queries per second during peak times while maintaining a maximum response time within 50ms.

Unlike existing graph databases optimized for static data or depth-first search (DFS), S2Graph excels at frequent data changes and BFS based on partial data. Additionally, it adopts an architecture considering both push and pull models to support real-time updates and dynamic ranking logic for viral effects.

Architecture and Scalability

S2Graph utilizes the Storage-as-a-Service concept, leveraging HBase as the graph data store to ensure performance and scalability. The storage layer is designed independently of physical implementations, allowing the use of other databases such as MySQL. The complex manual sharding architecture prior to adoption was improved to a clean, infinitely scalable structure after introducing S2Graph. Through core improvements, QPS increased from 20,000 to 65,000, and response times improved from 100ms to under 50ms.

Open Source Project Status

In November of last year, S2Graph was selected as an Apache Foundation Incubator Project and renamed Apache S2Graph. Source code migration and documentation work are currently in progress, and it can be easily installed and tested on a single machine using Vagrant and VirtualBox.