LG AI Research: LLM Efficiency Research Through the Lens of ACL 2024

Recently, LLMs (Large Language Models) have shown outstanding performance in various fields such as natural language understanding and generation, but as model parameters grow to hundreds of billions, they face the problem of rapidly increasing training and inference costs.

To address this problem, ACL 2024 intensively covered various algorithm and architecture research aimed at optimizing resource usage while maintaining model performance.

A representative study, Meta's LayerSkip, presents a methodology for speeding up LLM inference. This approach is based on the observation that even the lower layers of a model can produce sufficiently accurate predictions, and it leverages Early Exit and Self-Speculative Decoding techniques.

The main approaches of LayerSkip are as follows:

• Applying Layer Dropout and Early Exit Loss: During training, different dropout probabilities are applied per layer, and a direct loss function is introduced so that lower layers contribute better to the language modeling task.
• Efficient Inference: During the inference process, when the model predicts a result early, it terminates without going through the remaining layers, reducing memory usage and increasing speed.