Templar Publishes Fault Tolerance Simulation for Pipeline Parallel Training

Templar published simulation results for a stage skipping technique that enhances fault tolerance in pipeline parallel training on its distributed pre-training platform, Crucible. This technique skips computations for specific pipeline stages that go offline, allowing healthy stages to continue processing tokens and thereby reducing recovery wait times.

Simulation Setup and Results

• Environment: 178M parameter model, 8 replicas, 4 stages per replica
• Failure Conditions: 1% replica failure probability per global step; stages are removed for 6 global steps upon failure
• Performance: Validation loss remains at a similar level compared to the no-failure baseline

Effect of Fixed Projections

Fault tolerance is further improved when fixed projections shared between layers are used alongside pipeline compression. This is hypothesized to be because shared projectors align representations at stage boundaries, reducing the disruption caused by stage skipping, though this remains a hypothesis.

Implications

This research suggests the possibility of training on a broader pool of computing resources, including unstable workers or spot instances. However, it explicitly states that these are simulation results of the impact of stage failures on training, rather than measurements of physical worker replacement or actual cost savings.