Building a 24×7 Instant Response System for AI-Based Online Services

LotteON has operated a monitoring environment using WhaTap, Datadog, Grafana, and others, receiving real-time alerts via Slack. During weekday business hours, personnel checked alerts and responded, but nights and holidays required 24-hour monitoring staff for swift action.

Monitoring staff relayed incidents to Dev, Infra, DB, and Sec domain owners via Slack and phone based on approximately 100 notification criteria. However, under the 'You build it, you run it' principle, technical issues in each domain and incidents during nights and weekends decreased, increasing the potential for automating operations staff.

Since the existing Slack-based monitoring was working stably, it was determined that combining it with automated outbound calling could automate operations. To this end, the direction of building a system for 24-hour instant notification of online service incidents using AWS's AI contact center service, Amazon Connect, was considered.