StarRocks Operations: Isolating Multi-Tenant Workloads with Resource Groups

To handle both service lookups and analytical queries on a single platform, StarRocks was adopted as a real-time OLAP engine. But as service, batch, ingestion, and dashboard queries all piled up on a single cluster, the core operational challenge became not average performance but "which queries should be protected first."

The clusters in operation had very different characteristics. The service-facing cluster handled roughly 69 qps average over 24 hours and 87 qps average over a week of lookup traffic, while the monitoring/batch cluster handled about 20 qps average over 24 hours alongside heavier batch jobs. In this environment, it was necessary to classify workloads using Resource Group and design CPU priorities accordingly.

The basic strategy was cpu_weight. Since CPU is only split proportionally when contention occurs, it allows smooth adjustment of relative priorities among workloads like service, batch, and dashboard in a typical multi-tenant environment.

• service_wg: for service queries, highest priority
• batch_wg: for server batch jobs, medium priority
• dashboard_wg: for lookup tools like Grafana, Tableau, and Redash, lowest priority

However, when service-level latency SLAs mattered, exclusive_cpu_cores was needed. This option reserves physical CPU cores exclusively, internally binds threads to cores, and even separates a dedicated ThreadPool to eliminate contention with the shared pool. In the Toss Shopping case, cpu_weight was first adjusted to protect service queries, but when response time spikes kept recurring, the service account was separated into its own group and exclusive_cpu_cores was applied. As a result, response time spikes disappeared even when batch workloads overlapped.

In classifier design, matching based on user or db was the most stable. Since the db condition carries particularly high weight and is almost always prioritized, it was important to design rules so that different ones don't overlap. At the same time, what Resource Group directly manages includes regular queries, INSERT INTO, and Broker Load, while Routine Load and Stream Load are not directly controlled.

Controls beyond CPU were also used together. concurrency_limit limits the number of concurrently running queries within a resource group, queuing the excess, and big_query forcibly terminates queries that exceed thresholds for CPU time, scan rows, or memory usage on a BE-node basis. mem_limit is a cap per group rather than a reservation, so the sum across multiple groups can exceed 100%, and spill_mem_limit_threshold is just the criterion for starting to consider spilling — actual behavior also requires enable_spill, spill_mode, and operator support to align.

The most important pitfall in operations was the Docker deployment environment. When running BE and CN as containers, --cpus alone is not enough; --cpuset-cpus must also be set for bind_cpus and cpu_borrowing to work as expected. Without cpuset, the structure for borrowing dedicated cores breaks down, which can significantly reduce the efficiency of exclusive_cpu_cores.

In an actual deployment case, setting exclusive_cpu_cores=50 on load_wg limited CPU occupancy to about 60% on a 92-core BE, but INSERT job time increased to about 380–457 seconds. There was a trade-off of slower jobs as dedicated cores were reduced, but it prevented the entire cluster from stalling. The key was to start with cpu_weight and escalate to exclusive_cpu_cores only when needed, then layer classifier, concurrency_limit, big_query, and mem_limit on top to progressively control multi-tenant workloads.