Toss Securities: Building a GPU-Native Cluster Beyond the Limits of GPU-Awareness

The Data Infra team and ML Platform team at Toss Securities shared their experience building a GPU-native cluster based on Kubernetes. The previous GPU-aware state was limited to Kubernetes recognizing and allocating GPUs as resources, but actual operations revealed clear limitations, such as the inability to control VRAM, discrepancies between node status and GPU status, and batch inefficiencies caused by integer-unit allocation.

GPU Readiness and Operational Stability

Toss Securities distinguishes between the Ready status of a node and the Ready status of individual GPUs. The existing nvidia-validator only checked the exit code of nvidia-smi during driver verification, which led to false positives where individual GPUs with ERR! status or Xid errors were incorrectly judged as normal. To resolve this, a separate DaemonSet was introduced to comprehensively validate nvidia-smi output, PCI device count consistency, and the occurrence of fatal Xid errors.

Additionally, DCGM Exporter is used to continuously monitor internal GPU states. VRAM usage (FB_USED/FREE/TOTAL), compute unit utilization (SM_ACTIVE), and health status (HEALTH_STATUS) of detailed items such as DRIVER, MEM, and NVLINK are collected as Prometheus metrics. Notably, since Xid errors are not observed as Kubernetes events, they are utilized as operational signals for isolation and recovery decisions rather than simple logs.

Resource Granularity and Batch Optimization

To enhance the flexibility of GPU resource allocation, MIG (Multi-Instance GPU) technology was applied. MIG Manager automates MIG configuration based on ConfigMap, reducing the burden of manual management. Furthermore, to address the issue where the default distributed allocation method of the NVIDIA Device Plugin scattered MIG slices across multiple GPUs, making it difficult to secure Full GPUs, a Packed Allocation method that fills one physical GPU first was proposed and reflected upstream. This is provided as the sharedDevicesAllocationPolicy setting starting from v0.20.0.

New Equipment Onboarding Procedure

Before introducing new equipment such as the Blackwell series or returning nodes to the cluster, a GPU preflight procedure is conducted. Nodes must pass five stages of gates: standard specification consistency check, hardware burn-in test, NVLink bandwidth measurement, CUDA runtime verification, and small LLM serving workload testing. Only nodes that pass all verifications are assigned the preflight=passed label and uncordoned to allow service deployment.