AI Briefing#distillation
The latest AI and developer news about #distillation, with the original source and a short summary.
The latest AI and developer news about #distillation, with the original source and a short summary.
TIPSv2 significantly boosts patch-text alignment through iBOT++, Head-only EMA, and multi-granularity captions.
Frontier models are harder to prevent **distillation** of than expected, and large-scale pretraining can easily collapse from small **bias** and **causality** violations.
A dataset of 100,000 CoT samples for fine-tuning local reasoning models has been released.
Spotify pre-generated 1.4 billion reports based on listening data from 350 million people.
By leveraging high-quality reasoning data, a 15B reasoning model was distilled into a 2.1x faster Mamba hybrid without performance degradation.
SD3.5-Flash is a lightweight distillation framework that produces high-quality images in just a few steps.
Kanana Nano-2.1B maintains 93% of Kanana Essence's Korean performance despite a 26% size reduction.
Kakao's Kanana Essence, trained on 3T tokens, achieved higher benchmark averages than llama-3.1-8b, while Kanana Nano maximized efficiency through pruning and distillation techniques.