Kakao Bank Rebuilds Notification Sender Handling Hundreds of Millions of Daily Messages: An Architecture Balancing Performance and Simplicity

The Kakao Bank Notification Experience Engineering team rebuilt the notification sender (UMS) that processes hundreds of millions of notifications daily, achieving two goals: performance optimization and simplification of the developer mental model. The existing system suffered from Head-of-Line Blocking due to the lack of Thread Pool usage and a single-queue structure, with 80% of P99 Latency consumed by waiting time.

Isolating Complexity via Service/Core Layer Separation

The new architecture encapsulates complex performance logic in the Core Layer and provides a simple interface in the Service Layer, allowing developers to focus solely on business logic. Service Layer developers only need to implement a total of 3 methods from the RequestHandler and Sender interfaces, while the Core Layer handles complex controls such as retries and timeouts. This eliminates the need to modify Service Layer code when replacing Sender vendors or optimizing the Core Layer internally, significantly improving scalability and maintainability.

Age-Based Work-Stealing and Lock-Free Implementation

In the Core Layer, Age-Based Work-Stealing was introduced to achieve high throughput. A central manager scans the queue every 100ms and distributes tasks with long waiting times (Age) to idle Workers, significantly reducing the waiting time of fast tasks queued behind slow ones. For example, a task that waited for 2000ms saw its processing time reduced to 600ms.

Additionally, a Lock-Free structure was adopted to handle over 10,000 Steal operations per second. By combining CAS (Compare-And-Set) and Barriers to determine atomic splits, and using Zero-Copy techniques to split queues by adjusting indices without copying arrays, memory allocation and GC pressure were minimized. Fixed-size Arrays were used to maximize CPU cache efficiency, and object overhead was eliminated through Kotlin inline classes and template caching.

Results and Implications

The rebuild achieved a throughput of over 10,000 TPS and P99 Latency under 200ms. The time required to add new channels was reduced, and developers can now develop notification sending features without understanding concurrency or performance optimization logic. This is evaluated as a successful case of a structural approach that increased the overall development speed of the team by hiding technical complexity and providing simple interfaces.