Optimizing Trino Resources on YARN

By re-configuring the resource configuration of a Trino cluster deployed on YARN, the team increased available memory for queries by about 40% without adding any hardware.

PDL (Place Data Lake House) provided Hive and Trino as engines for querying Iceberg tables, and as Trino usage grew rapidly, a resource review became necessary. Since Trino excels at interactive queries and is widely used for AdHoc, BI, and exploratory analysis, the team chose to use existing resources more efficiently rather than simply scaling up.

In the production environment, Trino was deployed as a YARN Service to leverage the benefits of the Hadoop ecosystem. However, unlike a model where resources are only used when a request comes in, Trino requires Workers to always occupy resources, so a dedicated queue had to be created separately and node resource allocation had to be finely tuned.

Resource optimization proceeded along three axes.

• Taking the Hadoop daemon processes into account, memory available for YARN applications was limited to 100GB after subtracting a margin from total node memory for operational stability
• Considering the AM Container's occupancy, one node was assigned to host the Coordinator, and Workers were placed on the remaining nodes
• Workers were sized as large as possible, designed so that a single Worker could maximize the use of a node's resources

The problem was that when Workers were configured to fill nodes completely, the last Worker would fail to start. The cause was YARN's RESERVED Resource mechanism, where an application that couldn't be allocated resources remained in a reserved state, preventing the last Worker from obtaining the resources it needed. To avoid this, the entire cluster's resources had to be recalculated by additionally accounting for the resources of one Worker.

As a result, the final configuration was rearranged to Coordinator 56GB Memory / 29 vCores and Worker 98GB Memory / 47 vCores, and it was confirmed that all Workers started normally. To prevent recurrence, an Active Nodes Alert was also added, which triggers a notification if the configured number of Workers fails to come up.

After finishing resource allocation, Trino configuration values were also tuned. The key was to test and confirm which each setting applied to—Coordinator or Worker—before placing it.

• JVM Max Heap: Set higher than the official recommendation (70~85% of node memory), operating at up to 90% of allocated memory
• query.max-memory-per-node: Applied to Worker, adjusted so as not to exceed the sum of JVM Max Heap and memory.heap-headroom-per-node
• query.max-memory: Applied to Coordinator, limiting any single user from using more than half of the cluster's resources
• spill-enabled: Enabled only on the Dev Queue to mitigate query failures via disk spill when memory runs low
• spiller-spill-path: Set to a fixed path on Workers to simplify spill file management

As a result, even within limited YARN resources, more memory was secured for direct use in Trino's query processing, and both per-user resource skew and out-of-memory failures were reduced. However, due to the AM Container and RESERVED Resource, it was not possible to utilize 100% of total resources, and the team plans to also review the possibility of running Trino on Kubernetes going forward.