Self-Speculative Decoding Accelerates Inference

Self-speculative decoding is a technique that, unlike traditional approaches using a separate draft model, generates tokens using the early layers of the same model and then verifies them in later layers.

This approach increases text generation speed and significantly reduces computational latency and memory usage. It is especially advantageous for efficiently deploying large-scale models in small GPU environments.

To maximize performance, a special training method (LayerSkip) that optimizes the output of the early layers is needed, and it can be implemented via the assistant_early_exit argument in Hugging Face's transformers library.