Kakao Releases Technical Report for Proprietary LLM 'Kanana' and Open-Sources Nano 2.1B

Kakao released a technical report detailing the full architecture, training process, and performance of its proprietary AI model family 'Kanana'. This announcement validates the completeness of the language model lineup consisting of Kanana Flag, Essence, and Nano, and includes the open-source release of Kanana Nano 2.1B to contribute to the research ecosystem.

Securing Global Competitiveness and Training Efficiency

Kakao applied various training techniques to achieve State-of-the-Art (SOTA) performance within limited resources. As a result, the top-tier model Kanana Flag 32.5B achieved high performance while reducing training costs by over 50% compared to third-party models. Specifically, it showed performance similar to global models on English benchmarks (MMLU, MT-Bench) and demonstrated overwhelming processing performance compared to competing models on Korean benchmarks (KMMLU, KoMT-Bench).

Staged Pre-training and Model Scaling Strategy

Kakao maximized training efficiency through a Staged pre-training approach. After first training models of 8B and 26.8B sizes, the model lineup was expanded based on them as follows:

• Lightweighting: Kanana Nano 2.1B was built by applying pruning and distillation to the 8B model.
• Scaling: Kanana Essence 9.8B and Kanana Flag 32.5B were developed by applying Depth Upscaling (DUS) techniques to the 8B and 26.8B models.

Through this strategy, high-performance models were secured at less than half the training cost of similar-sized global models.

Open-Source Release of Nano 2.1B for On-Device Use

To enhance utility for researchers and developers, three versions of Kanana Nano 2.1B—base, instruct, and embedding—were released as open source. This model is designed to run smoothly in on-device environments. While it may have limitations in complex reasoning or high-difficulty math problems, it is expected to have high utility across various application fields. In the future, Kakao plans to enhance reasoning capabilities and multimodal features by incorporating Reinforcement Learning (RL) and Continual Learning.