A story of repeated experimentation and improvement before deploying a new service

VSMS Deployment and Performance Goals

Market Kurly's Virtual Stock Management System (VSMS) was deployed live on August 30, 2020. This system was prepared for about 8 months by the planning, development, and DevOps teams, with the admin API, customer-facing API, and batch jobs as its main features. Since the stock lookup API is called on every page, high performance was required, so the team set a target TPS of 1200 — 4 times the expected Peak TPS of 300 — and conducted various performance tests.

Deadlock Resolution and DB Bottleneck Analysis

A deadlock issue discovered near the end of development was resolved with a 3-line code change that unified the lock order for A and B. Repeated performance testing confirmed that DB I/O was the biggest bottleneck. To address this, they applied experimental code that consolidated processing into a single class and batched commits together. They also attempted to reduce RDB connection occupancy by separating the quantity change log from the main DB into MongoDB.

Performance Test Results and Deployment

In the performance test after separating the logs, the system processed about 61.61 million requests over 11 hours 45 minutes, recording an average of over 1,450 TPS. After further work, VSMS stably maintained over 1500 TPS, achieving the target performance. Following a background deployment on August 12, 2020, it went into full operation on August 30. The experience reaffirmed the importance of simple code that sticks to the basics, rather than flashy solutions.