AI BriefingLG AI Research 435
Key point
LG AI Research presented research on the model compression technique LeSS and LLM structured pruning at an ICLR 2024 workshop.
Details
LG AI Research's DI Lab shared research achievements in model lightweighting, which reduces model size and computational complexity while maintaining performance, at an ICLR 2024 workshop.
The first study, LeSS, is a technique that efficiently integrates Filter Pruning and Low-rank Decomposition. To solve the problem where existing methods showed limited performance gains when combining these two techniques, the team proposed a new SGD-based learning technique called Threshold Learning.
LeSS enables end-to-end learning through a differentiable module, and achieved higher performance than existing models while significantly reducing FLOPs across various models such as ResNet56 and MobileNetV2.
The second study, Sheared Llama, addresses Structured Pruning of LLMs. To solve the problem where existing pruning methods create irregular model structures that cause inference overhead, this approach compresses the model to satisfy a predefined Target Structure.
This research learns masks at various Granularity levels, from global elements such as layers and hidden dimensions to local elements such as attention heads, and uses the Lagrange multiplier method to directly impose constraints on the shape of the model.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.