Yeogi-eottae Builds APM Scatter Chart with OpenTelemetry and ClickHouse

Yeogi-eottae built an APM scatter chart using OpenTelemetry and ClickHouse to address the issue where existing aggregated metrics lost distribution information for individual requests. The solution branches off the existing OTel/Grafana pipeline to store individual span data from all traffic in ClickHouse and visualizes it using a custom Grafana panel.

Scatter Chart Design and Data Storage Strategy

Each point in the scatter chart represents an individual span. The X-axis uses the end time (timestamp + duration) instead of the start time to allow identification of current system latency. Readability is improved by displaying entry-point spans by default while hiding sub-calls.

ClickHouse's column-oriented structure improves disk I/O efficiency by reading only the columns required for queries and offers high compression rates. Local benchmarks show that with ZSTD compression, it occupies approximately 1/10 (90% reduction) of the original storage space. Due to the characteristics of the MergeTree engine, frequent small INSERTs cause performance degradation, so bulk batch processing is preferred. Physical sorting and indexes are defined via ORDER BY.

OpenTelemetry Collector Integration and Filtering

Data is transmitted using the official ClickHouse Exporter (beta). The existing tracing path and the scatter chart path are separated to prevent traffic volume distortion caused by sampling. On the new path, batch timeout is set to 2s and send_batch_size to 2048 to prevent partition explosion.

Filtering is based on user-perspective processing units and response time semantics. Among INTERNAL spans, schedulers are excluded, as are non-HTTP CLIENT spans and health checks. The filter processor in the OTel Collector is used to set DROP conditions, with error_mode: ignore to prevent pipeline interruption.

Schema Optimization and Index Configuration

Raw data (raw, TTL 1 day) and query data (transactions, TTL 7 days) are separated. Map access and type conversions are pre-processed via Materialized Views. LowCardinality(String) is applied to columns with few unique values to improve storage efficiency, and FixedString(32) is applied to trace_id.

ORDER BY is configured as (service_namespace, service_name, toUnixTimestamp(timestamp), trace_id) to optimize prefix filtering. Skip Indexes such as bloom_filter and minmax are used for searching columns not included in the sort key. For full time-range queries, minmax skip index on ts_end performs granule pruning to reduce storage and INSERT costs.