Efficient Large-Scale Distributed Training of Superb AI's Vision Foundation Model 'ZERO' with Amazon SageMaker HyperPod

Superb AI trained the vision foundation model ZERO (Zero-shot Object Detector) by selecting 4 million images meaningful to real industrial sites from about 1 billion raw images. ZERO's core capability is multimodal grounding, which precisely detects object locations from text prompts or image-based visual prompts, enabling zero-shot detection of diverse defects and objects in open-world environments without additional Fine-tuning.

For large-scale training infrastructure, Superb AI chose Amazon SageMaker HyperPod. While existing clouds tend to be single-node focused or locked into annual plans, HyperPod allows flexible instance sizing and comes with high-performance networking such as EFA (Elastic Fabric Adapter) built in, making it advantageous for both distributed training and cost efficiency. This was especially suitable for a development cycle where GPUs aren't used constantly, allowing large resources to be used only when needed.

The training workflow was organized in the following order:

• Secure the large-scale training period first via Training Plan reservation
• Set the timeline for data preparation and hyperparameter tuning
• Run rehearsals on low-cost GPUs to verify data integrity and script bugs
• Once ready, release the rehearsal instances and run the actual training on a high-performance GPU cluster

Data preparation was also a key optimization point. Opening the raw data as individual files directly in Hugging Face Datasets would take about 30 hours for Arrow file conversion, but Superb AI pre-split the data into about 1,200 shards of 1G each in an Arrow-based dataset format and uploaded them to Amazon S3 in advance. With this approach, the data was ready to read within minutes, significantly reducing the data loading bottleneck.

Cluster configuration and FSx integration were verified in advance during the rehearsal stage. Two g6e.8xlarge instances were used as compute nodes to confirm that multi-node training worked correctly, and the mapping between FSx for Lustre's data repository path and file system path was carefully configured. Since more data can lead to longer metadata processing and initial loading times in FSx, upfront design was important.

The rehearsal was not a simple test but a stage that reduced the cost of failure in the actual training. Minor but critical issues such as wandb login errors and checkpoint directory ownership problems were caught in advance, and pre-loading S3 data into FSx before the Training Plan started saved several hours.

When transitioning to the actual training, the rehearsal instances were removed, provisioning_parameters.json was modified, and the cluster was reconfigured to match the newly reserved instance group and Target Instance Count. After the new instances came up, scontrol reconfigure was used to clean up the Slurm and Pyxis plugin states, and actual training was then carried out on the high-performance instances. Training was able to start within about 30 minutes after the Training Plan began, and thanks to the rehearsal, nearly the entire reserved period could be used for training.

As a result, HyperPod simultaneously delivered development productivity, cost efficiency, reduced training time, and improved stability. Superb AI was able to reduce the burden of infrastructure operations and focus on advancing model performance, and ZERO is also offered in a form that can be directly subscribed to and deployed via AWS Marketplace.