Kakao Releases Kanana-1.5 Reasoning Performance Enhancement Research… Proposes SFT and RL Optimization Strategies

Kakao has released the research process for applying reasoning capabilities to the Kanana-1.5 model. It shares specific optimization strategies and experimental results from the Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) stages, which are core to existing LLM reasoning techniques, particularly emphasizing the importance of the SFT stage.

Key to the SFT Stage: Data Selection and Learning Rate

The research team introduced a Two-staged SFT strategy to internalize reasoning capabilities. This approach selects data using verify_score (accuracy) and CV (coefficient of variation, a metric for difficulty and learning potential).

• Stage I: Selects data of appropriate difficulty to learn basic reasoning patterns.
• Stage II: Focuses on high-difficulty data with high CV to maximize reasoning performance.

Additionally, it was confirmed that the learning rate (LR) is a key factor determining the performance ceiling. The highest performance on the AIME 2024 evaluation was recorded when sufficiently large LRs were used in both Stage I and II, while excessively small LRs caused performance degradation.

Optimization in the RL Stage: GRPO vs. PPO Comparison

In the reinforcement learning stage, DeepMath-62K was constructed from the DeepMath-103K dataset through difficulty filtering and deduplication. Questions that could be answered correctly without a reasoning process, such as multiple-choice or Yes/No questions, were excluded.

• Clipping Strategy: It was confirmed that relaxing the lower bound ($\epsilon_{low}$) to 0.99 and appropriately setting the upper bound ($\epsilon_{high}$) in the GRPO algorithm is effective for learning low-probability tokens.
• PPO vs. GRPO: Comparative experiments under identical conditions showed that PPO, which explicitly approximates the Value Function in the later training stages, achieved more stable and higher Rewards than GRPO.

Cross-Domain Learning and Generalization

It was observed that math benchmark performance improved even when reinforcement learning was performed solely on code datasets. This was confirmed through the Staged RL strategy, suggesting that learning code also acquires the reasoning capabilities necessary for solving math problems. Models trained in a code-then-math order recorded approximately 4 points higher on AIME 2025 compared to models trained only on math.