Yeogieotteo Event-Driven Unified Notification Platform Build Story Part 2. How Was It Implemented?

NotiHub is a centralized unified notification platform built to manage company-wide notifications in one place. To solve the operational risk caused by webhook dependency and the difficulty of grasping notification status, the team prioritized stability and monitoring efficiency over convenience.

The core structure is a 3-tier separation of Receiver - Kafka - Processor. The Receiver only handles JSON schema validation and key verification, then immediately publishes events to Kafka to absorb burst traffic, while the Processor handles the actual routing, rendering, and delivery. For traffic surges, Pods are scaled out using HPA (Horizontal Pod Autoscaler), and logical distribution is achieved through Kafka Consumer Group and internal sharding.

External touchpoints are divided into internal network and external network. The internal network was designed so migration could be done simply by swapping URLs without an auth token, while external networks like Jira and Slack API go through B2B Gateway for dual verification. A global Static Key and endpoint-specific Dynamic Key are checked together, and since the Static Key changes infrequently, it is read from the Secret Store at startup and kept in an in-memory cache.

Settings and templates are operated with a triple-layer cache structure.

• Lookups proceed in the order Local Cache → Redis → API/DB to reduce latency.
• If Redis fails, the local cache is used first, and in the worst case the API server is called directly.
• Exponential Backoff is applied to Redis connection issues, and configuration update failures are retried up to a maximum of 3 times.

For Slack delivery, both rate limit and message integrity are handled together. Slack Bot Sharding is applied so that the same channel is always handled by the same bot via bot_index = hash(channel_id) % bot_pool_size, and independent queues are maintained per channel so that even if a specific queue gets blocked, other queues continue processing. When a message error occurs, the Processor detects cases such as no_text, msg_too_long, and blocks_too_many and recovers by splitting and resending.

The Processor routes events to multiple channels, and after a condition engine evaluates rules like eq, neq, contains as well as time and holiday conditions, it renders the Slack message using Handlebars. Events with the same incident_id are grouped into a thread, and if there is no response, escalation proceeds up to 3 stages, processed in a 1-minute batch cycle. Kafka Lag-based HPA and Graceful Shutdown have not been implemented yet, but the plan is to advance these step by step while monitoring ingestion volume and processing metrics.