Real-time Comment Development Log (2): Verifying 10k Connections with a GoLang Client and Infrastructure Tuning

For Alex's comment service with 600k DAU, the goal was to stably maintain 10k connections and 2800 TPS per real-time comment server. The existing HTTP-based testing tool, nGrinder, was unstable in maintaining socket connections and handling the STOMP protocol, so a custom client was built in GoLang for its high resource efficiency to conduct tests.

Infrastructure and Network Tuning

The intermittent memory shortages and connection drops encountered during testing were identified as being caused by VIP TPS limits and the 'Noisy Neighbor' phenomenon in the VM environment. To resolve this, the maximum TPS of the VIP was adjusted, and the environment was migrated to physical servers (PM) to eliminate resource contention. Additionally, the OS-level fileDescriptor limit was raised to the kernel's allowed maximum, and Docker container settings were optimized to secure connection capacity.

Application and Memory Optimization

Neo and fastThread were used to analyze the internal structure of Spring WebSocket and identify deadlocks and blocking code. In particular, an issue where the DestinationCache class blocked multiple threads was discovered, and the logic was modified. Regarding memory management, it was confirmed that GC load increased significantly when the frequency of STOMP SUBSCRIBE/UNSUBSCRIBE operations was high. Accordingly, instead of using SynchronousQueue, the existing queue approach was maintained, but Xms, Xmx, and Xss values were finely tuned to prevent a sharp decline in request processing speed during GC events.