AI BriefingK-Merge: Online Continual Merging of Adapters for On-Device LLMs
Key point
We propose the K-Merge framework, which continually expands an LLM's capabilities without data even in storage-constrained on-device environments.
Details
When running LLMs in on-device environments such as smartphones, memory and compute resources are very limited. PEFT techniques such as LoRA (Low-Rank Adaptation) are used to address this, but as new tasks keep being added, problems arise such as running out of storage space and Catastrophic Forgetting, where existing knowledge is lost.
K-Merge, proposed by Samsung R&D, is a lightweight algorithm performed directly on-device without data. This framework operates through a two-stage pipeline.
- Distance-Based Adapter Selection: When a new adapter arrives, it calculates the weight parameter distance to existing stored adapters to find the closest "nearest neighbor." Merging similar adapters minimizes interference.
- History-Aware Dynamic Merging: Instead of simple averaging, it uses a Merge History Counter to adjust weights. This prevents older core capabilities from being overwritten by new updates.
Additionally, through the K-Merge++ method, which immediately performs merging when a highly similar adapter arrives even before storage is full, it supports more proactive and efficient model management.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Toss Doctor V2 Released: Maximizing QA Efficiency with LLM-based Automated Testing and Self-healing