Building Microservice Observability with Standards

As the number of services grew in a Kubernetes environment, the need to unify scattered monitoring tools and simplify problem tracing became greater. As a solution, we combined OpenTelemetry and SigNoz to build an observability system that handles Metric, Log, Trace (MLT) in a single flow.

The problems with the existing environment were that tools were scattered, each had to be learned and operated separately, and tracing the cause of failures took a lot of time. Having a standardized telemetry collection system can reduce vendor lock-in and lower operational costs while connecting to various backends.

The reason for choosing OpenTelemetry is that it unifies the core signals of observability and can link correlations between services through Context Propagation. The context contains a span ID and trace ID, and propagation is mainly handled by auto-instrumentation libraries, with manual control possible via the Propagators API if needed. The default propagator uses W3C TraceContext-based HTTP headers.

The collection layer was configured with the OpenTelemetry Collector. The Collector is vendor-neutral and handles data via Receive -> Process -> Export, reducing the need to operate multiple Agents separately with a single Collector.

• Receivers: Receive network input such as OTLP or collect data via scraping
• Processors: Process data using memory_limiter, filter, metricspan, batch, etc.
• Exporters: Send data to destinations such as OTLP, Kafka, Elasticsearch, etc.

Order mattered in configuring the pipeline. First, memory_limiter controls the Collector's memory, then filter reduces unnecessary data, metricspan links Spans and Metrics, and finally batch groups data together to improve transmission efficiency. The same receiver can also be fanned out to multiple pipelines, allowing for flexible design.

Application instrumentation was applied via Auto-Instrumentation. After installing CertManager and the OpenTelemetry Operator, we created an Instrumentation CR and added language-specific inject settings to the Deployment's template.metadata.annotations. For example, Java uses instrumentation.opentelemetry.io/inject-java: "true", and behavior was configured with environment variables such as OTEL_SERVICE_NAME, OTEL_TRACES_EXPORTER, OTEL_METRICS_EXPORTER, OTEL_LOGS_EXPORTER, and OTEL_EXPORTER_OTLP_PROTOCOL.

Initially, an open-source combination like LGTM was also considered, but we judged that complexity would increase since each component would have to be learned separately and the Helm Charts were also independent. Instead, we chose SigNoz to unify into a single backend, viewing its ClickHouse-based processing, intuitive UI, OTel integration, and active community as advantages.

Ultimately, the goal was to gradually converge on the OpenTelemetry + SigNoz combination instead of using multiple scattered monitoring tools. Challenges such as log and trace retention strategy and further sophistication of Collector and ClickHouse operations remain, but operational convenience and maintenance efficiency have clearly improved.