Olive Young Improves AI Pick Recommendation Card Quality and Format Compliance Simultaneously via NLL-Based Alignment Learning

Olive Young data scientist An Il-ho shared the advanced development process of the AI Pick recommendation card generation model. The existing SFT (Supervised Fine-Tuning) method struggled to simultaneously satisfy format compliance, naturalness, and contextual appropriateness, with a notable trade-off between character count limits and natural expression.

Quantifying Unmeasurable Quality

To measure naturalness and contextual appropriateness, NLL (Negative Log-Likelihood) from the Polyglot-Ko-1.3B model was used as a proxy metric instead of LLM-as-a-judge or human evaluation. Lower NLL values indicate higher quality, with format compliance evaluated via regex-based validators and quality via NLL, separated for assessment. This enabled the automatic construction of preference data (chosen/rejected pairs) without human labels.

Balancing Format and Quality via DPO and GRPO

While DPO (Direct Preference Optimization) training improved naturalness (NLL), it caused a sharp drop in format pass rates. To address this, different strategies were applied per task.

• WHO Card (Personalization): The formatmix technique, which mixes near-miss error data generated by GPT-OSS-120B, was applied to recover the format pass rate by 13.2%p while minimizing quality degradation.
• TPO Card (Timeliness): Due to numerous format fields, maintaining format compliance with DPO alone was difficult. After securing quality via DPO, GRPO (Group Relative Policy Optimization) with shaped rewards for differential penalties on format violations was used to further recover the format pass rate by 24.7%p.

Validating the Effectiveness of the NLL Proxy

Compared against approximately 2,000 blind human evaluations, the 1.3B parameter NLL model showed higher consistency with human evaluations (Kendall tau-b 0.145, AUROC 0.694) than larger models like Gemini 2.5 Pro or GPT-OSS-120B. This demonstrates that NLL is a valid signal of actual quality rather than circular logic.

Operational Implications

Retries or reranking at inference time cannot replace internalizing quality and increase computational costs by approximately 5x. Therefore, a hybrid configuration that improves quality through alignment learning and handles remaining format errors via validator-based retries represents the optimal balance. Format validators and NLL metrics can be reused as common assets for model version comparison and pre-deployment validation.