Adopting the LGTM Stack for DevOps

As the number of services running on AWS EKS with MSA grew to dozens, the time needed to pinpoint issues increased sharply. Engineers had to jump between multiple dashboards and per-environment monitoring setups, and operating costs grew along with it.

To solve this problem, the team looked for an open-source observability stack that was Kubernetes Native and supported Object Storage, and ultimately chose LGTM. The goal was to connect metrics, logs, and traces in a single view, without increasing operational complexity as environments multiplied.

For Metrics, they first moved from CloudWatch to Prometheus. System visibility improved, but as environments increased, dashboard and alert configurations had to be repeated, and as metrics grew, Prometheus's resource usage also increased. In the end, they switched to Prometheus Agent mode and adopted a structure that sends data to Mimir via Remote Write.

The reasons for choosing Mimir are as follows.

• 100% Prometheus compatible: Remote Write, PromQL, and even alerts can be used as-is
• Horizontal scaling: scalability secured through a cluster architecture
• Object Storage support: can use S3, GCS, Blob
• High availability: supports replication structure
• Multi-tenancy: data isolation possible per tenant

Environment separation was handled with external_label, allowing a single dashboard and query to be applied across multiple environments as-is. For example, a common dashboard is maintained by simply changing the label, as in node_memory_MemTotal_bytes{env="$Environment"}.

For Logs, they switched from the existing EFK to Loki. Since most usage was log search and some alerts and dashboards, they went through PoC and VoC and concluded there would be no UX loss. Loki indexes only metadata, so it has good storage efficiency, and using Object Storage reduces the burden of cost and capacity management.

The key benefits of Loki are as follows.

• Object Storage support
• Real-time log tracking via Live tail
• Seamless integration with Grafana, making it easy to move between metrics, logs, and traces
• Multi-tenancy support

Since log label design is important for performance, they also referenced Loki-related label best practices. Also, rather than drawing dashboards directly from logs accumulated in Loki, they used Recording Rules to convert them into time series stored in Mimir before building dashboards, avoiding performance issues.

For Traces, collection began after adopting Istio. Jaeger and Zipkin were also considered, but they chose Tempo, which supports a lighter, Object Storage-based storage backend. Since the Istio gateway alone could not fully trace the entire call flow, trace generation and propagation were also added at the service level.

Since most of the backend is built with Spring Boot, they used spring-cloud-sleuth for 2.x and micrometer tracing for 3.x. On top of this, they added the OpenTelemetry Collector to filter out unnecessary data and inject needed attributes, and linked it with Loki to enable navigation between Trace -> Log and Log -> Trace.

Ultimately, the core of adopting LGTM was not simply replacing tools, but tying together service topology, RED metrics, logs, and traces into a single operational experience. The biggest achievement is building a structure that reduces repetitive work as environments grow and finds the root cause of failures faster.