LG AI Research Introduces K-EXAONE

LG AI Research developed K-EXAONE, a large-scale AI model built on proprietary technology to strengthen Korea's AI competitiveness. The model focuses on securing global-level performance while maximizing resource efficiency.

1. Cost Reduction through Efficiency

• Mixture of Experts (MoE) Architecture: Although it has 236 billion parameters, it activates only 9 out of 128 experts, using only about 23 billion parameters in practice, thereby improving resource efficiency.
• Multi-Token Prediction (MTP): Instead of the conventional autoregressive approach, it introduces a structure that predicts multiple next tokens simultaneously, improving inference speed by about 1.5x.
• Tokenizer Optimization: By expanding the vocabulary to 150,000 entries and applying the SuperBPE strategy, it improved tokens-per-byte by about 30% compared to before.

2. Systematic Training and Performance Enhancement

• Pre-training: The model is trained in stages on foundational knowledge, specialized knowledge, and reasoning ability, and is specifically designed to learn logical flow using Thinking Trajectory data.
• Context Extension: It supports up to 260,000 tokens, using a 'rehearsal dataset' to prevent performance degradation during long-context training, and validating this with the NIAH (Needle-In-A-Haystack) test.
• Post-training: Through tool-use learning in virtual environments (SFT) and Reinforcement Learning (RL) across various domains such as math and coding, the model's practical problem-solving ability was maximized.