How Kurly Monitors Orders

Limitations of the Legacy System and Background for Improvement

Market Kurly's development team faced monitoring limitations in their existing Legacy system due to rapid growth. When order volume was concentrated, SQL queries became complex, causing search failures, and joins across 6 or more tables plus subqueries frequently led to timeouts while waiting for responses. In addition, business logic and view logic were mixed together in a single file of over 2,000 lines, making maintenance difficult.

Architecture Transition Based on Kafka and the ELK Stack

The development team transitioned to an approach where data is loaded into Kafka when an order domain Event occurs in the Application, and a dashboard is implemented using the ELK Stack (Elasticsearch, Logstash, Kibana). This reduced the risk of data loss inherent in the existing file logging method, and leveraging the in-house Kafka-based platform improved development efficiency. Kafka is a distributed streaming platform that guarantees high throughput and data consistency.

Implementation Process and Operational Results

Data is sent to Kafka using the php-rdkafka client, and because of the asynchronous nature of the transmission, $producer->flush($timeout) is called to guarantee completion of transmission and prevent data loss. After deployment, real-time and cumulative order statistics can now be queried through the Kibana dashboard at a speed of around 1 second. After the initial deployment, some data consistency issues were discovered, so an additional ack (acknowledgement) option was configured, and idempotency processing based on log ID was applied to address data duplication and loss issues.