How Yeogieotti Established Observability Right-Sizing Criteria

Yeogieotti Company operates a self-hosted Observability stack with OpenTelemetry and Grafana LGTM, and newly established Right-Sizing criteria to re-fit Pod resources. A situation where nodes appear full may differ from actual usage, and conversely, if requests are set low, this could lead to OOMKill or CPU Throttling.

The criterion was P95, not the average or maximum value. 3-month data dilutes recent changes and 2-week data overweights event impacts, so 1 week was set as the default measurement period, using 5-minute sampling. When needed, 1-month data was cross-reviewed to filter out outliers.

For memory, container_memory_working_set_bytes was compared with kube_pod_container_resource_requests, and for CPU, container_cpu_usage_seconds_total was examined together with CFS throttling metrics. The formula was organized as Appropriate Request = P95 usage / target utilization rate, and if P95 memory is 250Mi, the appropriate Request at an 80% standard becomes approximately 312Mi.

Components were classified differently based on their behavior.

• Stateless: Components like Distributor, Query-frontend that remain stable even with a low buffer
• Stateful read: Components like Store-gateway, Querier that have latency impact but low risk of loss
• Stateful write: Components like Ingester where OOM before flush directly leads to data loss, requiring the largest buffer
• Burst pattern: Components like Compactor that normally wait but use large amounts of memory during jobs

In line with this classification, Ingester was given ample headroom equivalent to a +50% buffer, while Stateless was applied a lower buffer. Grafana's Mimir sizing recommendations, AWS Compute Optimizer's Balanced profile, and GKE VPA's conservative buffering were also referenced, but the final criteria were differentiated based on each component's peak pattern and failure impact.

Application started with Stateless, proceeding with 3-7 days of validation at each stage. As a result, Observability infrastructure resources were reduced without any increase in OOMKill or Throttling, and this was subsequently incorporated as a regular quarterly review item.