AI BriefingLG AI Research Blog
Key point
A look at efficient training methodologies and the latest technology trends for foundation models, which serve as the basis of LLMs.
Details
Large language models (LLMs) are evolving beyond simple assistants into a stage where they reason on their own and use tools. LG AI Research has contributed to these technological advances through the EXAONE series.
The Foundation Model, which is at the core of LLMs, must take both post-training and inference efficiency into account from the design stage. In particular, the Mixture-of-Experts (MoE) architecture is drawing attention as a way to scale up model size within limited resources.
Unlike Dense models, which use all parameters, MoE is a Sparse model architecture that reduces computational cost by activating only specific Experts depending on the input token. Recently, DeepSeekMoE has maximized efficiency by subdividing experts and introducing the concept of a 'Shared Expert.'
Low-precision training techniques such as FP8 are also being adopted to increase training speed and efficiency. DeepSeek-V3 minimizes performance degradation while improving training efficiency through a Mixed Precision Framework that uses FP8 for computation-critical parts and FP32 or BF16 for parts that require numerical stability.
