Yeogieotae's Journey Building an Event-Based Unified Notification Platform Part 1. Why Was It Built?

Good notifications are not simply a feature that sends a lot of messages, but a way to accurately deliver them to the right recipients and channels for a purpose, so that reading them immediately leads to the next action. As the service grew, notifications that were "fine to check or fine to ignore" piled up, and the problem of important signals getting buried grew bigger.

The existing Slack Integration was easy to set up and allowed immediate use of standard formats, but from an organization-wide perspective, it offered low flexibility. The ArgoCD notification channel mixed Healthy success notifications with urgent alerts like Degraded and Sync Failed, and GitLab also delivered MR and Pipeline notifications mixed together, conveying only results without cause information.

Initial analysis showed that about 94.48% of internal Slack webhooks were created with personal accounts. Although there was a process for issuing shared accounts, it wasn't sufficiently communicated during onboarding, and personal webhooks carried the structural risk of disappearing along with account deletion or personnel changes.

Notification policies were also scattered across multiple places, including gitlab-ci.yaml, Jenkinsfile, Grafana Alert, and the Slack webhook configuration screen. As people couldn't find where settings were located, they repeatedly abandoned existing channels and created new ones, resulting in 638 ghost channels with no recipients.

To solve this problem, Notification Hub (NotiHub) adopted a structure operated at the organizational level rather than by individuals. Based on membership information from GWS (Google Workspace), used by all members, team-level workspaces were created, and permissions were managed through the internal developer platform Devhub, converting operational ownership into an organizational asset.

The design direction had three pillars.

• As easy as Slack: Maintain the same interface as existing Slack webhooks so that changing just the URL makes it work immediately
• Separated on an event basis: Each system only announces that an event has occurred, while NotiHub handles routing and channel branching
• Delivered according to purpose: Distinguish between informational notifications and notifications requiring immediate response, and improve message quality through templates and routing rules

This approach eliminated the recurring cycle of code changes and deployments for simple channel changes or format modifications. ArgoCD separated the channel for deployment completion from the channel for incident response, and added the failure cause directly in the message body along with a deep link, so situations could be understood without going into the UI.

GitLab also separated MR notifications from Pipeline notifications and included review comment content directly in messages to reduce access costs. As adoption that started in some teams naturally spread across the entire department, it was confirmed that this is a notification platform used out of necessity rather than by force.

Currently, starting with the GitLab template, they are expanding templates for incident response, monitoring, and interactive notifications. The next Part 2 will cover the high-availability architecture and the event processing approach using Kafka and Redis.