Asynchronizing continuous batching

In synchronous continuous batching, the CPU and GPU work alternately, causing the GPU to sit idle 24.0% of the time. Hugging Face measured that, with 8K tokens, batch 32, and an 8B model, the total generation time was 300.6 seconds, and that eliminating CPU overhead could reduce this to about 228 seconds.

The solution is asynchronous batching, which overlaps the GPU computation of batch N with the CPU preparation of batch N+1. To do this, GPU work is placed on a non-default stream instead of the default stream, and H2D, compute, and D2H are separated and streamed in parallel.

• The default stream synchronizes with other streams, so it is not used.
• The non-default stream returns control to the CPU immediately, allowing CPU preparation and GPU computation to overlap.
• CUDA events are used to ensure compute starts only after H2D completes, and D2H starts only after compute completes.
• Simply splitting streams breaks ordering, so dependencies must be explicitly specified using events.

Hugging Face stated that this approach has been reflected in the continuous batching implementation of transformers.