Boosting Training Efficiency with Effective Training Time

Effective Training Time (ETT%) is the ratio of E2E wall time actually spent training on new data. Meta used this metric to measure waste in non-training periods, and broke bottlenecks down into 3 L1 metrics—Time to Start, Time to Recover, and Number of Failures—to drive improvements.

Analyzing fleet-wide ETT since H2'24, Meta introduced 40+ new techniques over the past few years. As a result, by the end of 2025, offline training ETT% was pushed above 90%.

The improvement areas fall into four main categories.

• Time to Start / Recover: Reduced trainer initialization and PT2 compilation time to lower startup delay and recovery time.
• Checkpoint Management: Reduced training interruptions caused by checkpoint saving, and minimized unsaved training time.
• Shutdown Time Optimizations: Moved model publishing for inference to CPU machines instead of GPUs, saving GPU time.
• Failure Reduction and Observability: Improved scheduling time and preemption job ratio, and reduced failure frequency through component-level observability and refined error classification.

Trainer initialization is broken down into device_init, process_group_init, preproc_creation, train_module_creation, init_plugins, pre_train, and get_first_batch_data. Here, two directions of optimization were key.

• Communication optimizations: Reduced unnecessary process group creation and inter-rank communication. For example, they reduced the repeated all_gather calls used to build shard metadata, and after broadcasting the sharding plan, had each rank construct its own global rank information from local metadata, reducing sharding time.
• Pipeline optimizations: Parallelized sub-stages that had no dependencies on each other. In particular, the delay caused by PT2 compilation and DPP warm-up waiting for the first batch of data was mitigated by using a fast batch to secure data earlier, allowing compilation to start sooner. This approach is especially effective for jobs with long data loading times, such as large Foundation Models.

PT2 compilation was improved along three axes.

• Reducing unnecessary recompilations: To reduce repeated recompilation caused by dynamic shapes, TORCH_COMPILE_DYNAMIC_SOURCES was created in H1'25. It allows marking parameters as dynamic without major code changes, and supports integers and regex as well, increasing flexibility.
• PT2 cache improvements: MegaCache bundled inductor, triton bundler, AOT Autograd, Dynamo PGO, and autotune settings into a single archive for sharing. This reduced remote server requests, decreased model setup time, and improved startup and retry stability. This work reduced average PT2 compile time by about 40%.
• Autotune config pruning: As the cost of exploring Triton kernel configurations grew, compile time was reduced by identifying the most time-consuming kernels and reflecting optimal runtime configs directly in the codebase.

Checkpoint management was another important axis. A checkpoint is a snapshot that saves model parameters, optimizer state, and progress, allowing resumption from the last save point even after a failure. However, the saving process can occupy GPU memory and time, blocking training, so Async Checkpointing and PyTorch Native Staging were used to reduce blocking time.

The key lies in systematically cutting time not spent on actual training while maintaining training quality and recoverability. As a result, Meta has used ETT% as an operational metric to drive training efficiency improvements across initialization, compilation, saving, recovery, and pre-shutdown phases together.