Real-time Exception Notification System Built with AOP and Redis

The existing exception notification relied on a 5-minute batch cycle and DB queries, which reduced real-time performance, and had the problem that notifications would stop if any one of the scheduler, batch server, or DB became unstable. There was also a major limitation where notifications would repeatedly flood if the same error persisted for a long time.

To solve this, a real-time notification system based on AOP + Redis was built. When an exception occurs, it first checks Redis for whether the same exception exists, and if it's a new exception, it is stored with a TTL of 5 minutes and a notification is sent. If the same exception already exists, the TTL is renewed to 30 minutes and the occurrence count is accumulated, and starting from more than 5 occurrences, unnecessary Slack notifications are blocked.

Failure response was designed using the Chain of Responsibility pattern.

• Redis normal, RDB normal: Exception history is managed in Redis and periodically synced to the RDB.
• Redis failure, RDB normal: Processing responsibility is handed over to the RDB, which determines duplication within the last 30 minutes.
• Both Redis and RDB failure: Since there is no basis for comparison, a notification is unconditionally sent when an exception occurs.

The exception history accumulated in Redis is synced to the RDB using the WriteBack pattern. The RDB serves as a safeguard to prevent duplicate notifications during Redis failures, and also functions as a record store for later failure analysis and system improvement.

During the sync process, using Redis SCAN as-is could cause duplicate returns during rehashing, and traversing a large number of keys could cause response delays. So the approach was changed to manage the keys to be changed as the value of a separate Redis key, reducing read performance impact and impact on other clients.