Toss Securities Oracle DB: Average CPU Normal but Commit Delays, NIC Interrupt Socket Concentration Is the Cause

• The cause of repeated commit waits of 0.1 seconds or more despite normal average CPU and metrics for each layer in Toss Securities Oracle DB was confirmed
• Delays occurred only when LGWR ran on CPU 28, and NIC interrupts concentrated on 8 physical cores, i.e., one socket, were confirmed as the bottleneck
• Two interconnect NICs were moved to different socket slots and bonding was changed to Active-Active to distribute interrupts across two sockets
• On another Oracle RAC where the same measures were applied, even on a day with 28% more concurrent users, the 10-minute average CPU utilization right after market open decreased from 36.6% to 17.4%, and the 10-minute interval maximum decreased from 42.5% to 21.9%

Structure of the delay

• Commit completion occurs only after LGWR writes the redo log to disk, and sessions are identified by the log file sync wait event
• NIC interrupts are processed before ordinary processes, stopping LGWR on the same core and delaying redo log writes
• Active-Standby bonding provides redundancy but normally carries traffic and interrupts on a single active port

Alternatives considered

• irqbalance redistribution was not applicable on that device due to NIC queue and CPU mapping constraints
• Pinning LGWR to a CPU on another socket with taskset was judged burdensome because it required manual configuration on every startup
• Moving the NIC slots and switching to Active-Active were chosen because they leave no settings to manage on every startup

Scope of application and additional observations

• The target was a 2-node, 4-socket Oracle RAC server with 32 physical cores and 64 logical CPUs through hyperthreading
• During the same peak period, the number of physical cores handling more than 1,000 interrupts per second increased from 8 to 15, logical CPUs from 16 to 31, and the Gini coefficient decreased from 0.80 to 0.67
• The average %system over the 15-minute peak interval on the originally problematic server decreased from 7.77% to 5.89%, even on a day with 33% higher load
• Execution time per execution decreased for 8 of 9 common SQL statements in AWR, and six key business SQL statements were shortened by 10.5% to 30.8%