A DevOps Engineer's Redis Testing Struggle - Part 1

During service enhancement and the MSA conversion process, an outage occurred following an API version update after introducing Redis. APM analysis confirmed a problem with the Look-Aside-Caching approach, where the absence of stored Key values caused load to concentrate on AWS RDS. Since there was no in-house standard Redis specification or a Sizing model based on performance testing, the team conducted testing to reproduce and verify the issue and to share the experience internally.

Test Environment and Tool Selection

Because AWS Elasticache does not allow receiving redis-benchmark commands from outside, and given the need to reproduce the issue and share it with the team, a simple API written in Node.js and Express was selected along with nGrinder as the load testing tool. The test environment consisted of cache.m5.large (Failover mode) and two EC2 c5.large instances. Since the existing PHP-based API uses the Strings type, code for handling Strings data was written using the NodeRedis client.

Load Test Results and Cause Analysis

In the first scenario, when load was applied with nGrinder at a Total Vuser count of 1,500, countless Errors occurred and TPS bottomed out. High CPU usage on the Read Endpoint was observed at the point of Key absence or Set-after-Expire and data checks, and when requests exceeded 200 users, Primary Endpoint CPU usage rose above 25%. This was determined to be a bottleneck in the Key-existence-check logic caused by slow command execution due to the Single Process nature of Redis or by simultaneous, frequent GET calls.

Optimization and Implications

In the second scenario, the Loop statement was removed and the logic was simplified to only perform Set and Value checks when the Key is null. As a result, CPU usage remained low even with long Strings values inserted, and the average TPS measured was much higher than in the initial test. According to the developer's remarks, when many clients call simultaneously and specific Key values change frequently, using File Cache or choosing an appropriate Collection can be more important than using Redis. It was confirmed that when using Redis, careful consideration of the data storage approach and preparation of failure response measures are essential.