The Process of Adopting OpenTelemetry

After adopting Kubernetes, services rapidly transitioned to MSA, making inter-service connections more complex, and the previous approach of fragmented collection via Filebeat and Metricbeat made CS analysis difficult. To solve this, the team reviewed adopting OpenTelemetry, which has effectively become the standard for observability, and configured it for the operating environment by referencing Saramin's application case.

The OpenTelemetry Operator uses Kubernetes' MutatingWebhookConfiguration and ValidatingWebhookConfiguration to inject Instrumentation into Pods and validate the OpenTelemetryCollector and Instrumentation CRDs. Installation was done via the opentelemetry-operator Helm Chart 0.105.1, and since Dynamic Admission Control—which operates during Pod creation and updates—requires TLS certificates, installing cert-manager was also recommended.

The Collector was used as the central component that collects, processes, and transmits telemetry data in a vendor-agnostic manner. The distribution was based on otelcol-k8s, and the pipeline was configured as follows.

• Receivers: otlp, filelog, hostmetrics, k8s_cluster, k8sobjects, kubeletstats
• Processors: memory_limiter, k8sattributes, filter, transform
• Exporter: sends to the Gateway Collector via otlp_grpc
• Mode: daemonset for collecting node-level metrics and file logs

RBAC was also configured. A ServiceAccount was created, and a ClusterRole and ClusterRoleBinding reflecting the permissions required by k8sclusterreceiver, k8sobjectsreceiver, kubeletstatsreceiver, and k8sattributesprocessor were linked; the serviceAccount was then specified on the Collector to complete deployment.

Application instrumentation was unified into a single Instrumentation CRD. The exporter endpoint was set to http://otel-collector.otel-namespace.svc:4317, propagators were set to tracecontext and baggage, and the sampler used parentbased_traceidratio with "1". After this, simply adding the annotation instrumentation.opentelemetry.io/inject-java: otel-namespace/instrumentation to a Deployment applies auto-instrumentation.

While the basic adoption was complete, the structure where file offsets and the exporter queue only reside in memory posed a risk of data loss during Collector restarts or network failures. To address this, the File Storage Extension was introduced to move storage to disk, continuing improvements to increase data persistence and integrity even during failure scenarios.