AI BriefingLG AI Research 436
Key point
At ICLR 2024, LG AI Research introduced LeSS, a new model compression technique that combines filter pruning and low-rank decomposition.
Details
Deep Neural Networks (DNNs) show excellent performance but consume massive computational resources and memory. In particular, with the emergence of LLMs (Large Language Models), the importance of Model Compression technology, which reduces model size while maintaining performance, is growing even further.
LG AI Research's DI Lab presented LeSS (Learning to Select a Structured Architecture over Filter Pruning and Low-Rank Decomposition) at ICLR 2024. Existing Filter Pruning and Low-rank Decomposition are based on different structural assumptions, making it difficult to effectively combine the two techniques, but LeSS integrates them through a learning-based approach.
The key features of LeSS are as follows:
- Learning-based Rank Selection: To select an appropriate rank without relying on heuristics, a new SGD-based learning technique called Threshold Learning was proposed.
- End-to-end Learning: Filter pruning and low-rank decomposition are simultaneously optimized through differentiable modules.
- Proven Performance: On the ResNet56 (CIFAR10) model, it showed excellent performance while achieving an 80% reduction in FLOPs, and on ImageNet it also demonstrated compression performance far superior to existing hybrid approaches.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Cerebras Serves Qwen 3.8 27B Model at 1,500 Tokens per Second