Config Builds a Large-Scale RFM Data Pipeline on Amazon EKS Spot

Config operates a preprocessing pipeline for a General-Purpose Robot Foundation Model that handles about 100,000 hours of action data and a new 20,000 hours per month of incoming data. The core task is to estimate robot-aligned action from human manipulation videos, then split and align them into episodes to build a trainable dataset.

Previously, the system used a single-queue structure based on Amazon SQS + AWS Lambda, but as users grew and large-scale batch processing increased, the bottleneck worsened. When jobs piled up in a single queue, sequential waiting occurred, and Lambda's concurrent execution limits, 15-minute execution limit, and On-Demand pricing structure made it unsuitable for large-scale workloads.

The new architecture combines RabbitMQ, KEDA, Karpenter, and Amazon EC2 Spot Instances on top of Amazon EKS. The key idea is to create an independent dynamic queue for each job, enabling parallel processing of multiple users' batches without interference and individually tracking the progress of each Job.

The infrastructure was separated according to the nature of the workload.

• On-Demand nodes: host core components like RabbitMQ, KEDA
• CPU Spot nodes: video preprocessing and postprocessing workers
• GPU Spot nodes: action labeling inference workers

To secure Spot capacity, they configured a diverse range of instance families, and designed Karpenter to detect pending pods and automatically provision nodes. RabbitMQ was configured with Quorum Queue to ensure message replication and durability, and retries and recovery were automated with x-delivery-limit, DLQ, and a 20-minute consumer timeout.

Auto scaling works via KEDA, which scales pods up and down based on RabbitMQ queue length. When the queue is empty, it scales down to 0, and when messages pile up, scale-up begins within 10 seconds. Karpenter then attaches nodes, flexibly scaling CPU/GPU workers.

Spot interruption handling was also built in multiple layers. Karpenter receives events and drains nodes, and when a worker receives SIGTERM, it NACKs the message being processed so it can be reprocessed. Messages are replicated in the Quorum Queue, and even if a worker terminates abnormally, it is automatically redistributed after the timeout.

The action labeling workload is divided into three stages: CPU preprocessing → GPU inference → CPU postprocessing, each with its own independent queue and ScaledObject. Thanks to this structure, processing of large-scale video and action data was parallelized, resulting in 70-90% cost savings compared to On-Demand, processing time reduced from several days to several hours, and 1,000+ pods running concurrently.