KakaoPay Insurance Resolves Loki OOM via Alloy Duplicate Collection Removal and Label Optimization

Details

OOM and 890 million log drops occurred due to duplicate collection from inactive Alloy clustering and position file loss
Duplicate collection and re-collection issues were resolved by enabling clustering, switching to loki.source.file, and using hostPath persistent storage
On a sandbox basis, Loki CPU decreased by 98% from 16.9 cores to 0.37 cores, and query P99 was shortened by 90%
On a production basis, resource efficiency improved, including a 52% reduction in Alloy RSS and a 93% reduction in Loki network RX
Active streams decreased by 78% by separating index labels from Structured Metadata and removing the app label hash
S3 bucket size was reduced by 86% by cleaning up DEBUG logs and shortening retention
Because multiple changes were deployed together, accurately attributing individual effects is difficult, requiring future measurement per unit of change

Key point