AI BriefingEdgeRazor: A Lightweight LLM Framework Supporting 1.58-bit Quantization
Key point
EdgeRazor is a lightweight LLM framework based on mixed-precision quantization and distillation that can be deployed across diverse hardware.
Details
EdgeRazor is a lightweight LLM training framework that can be deployed across diverse environments including mobile, edge endpoints, and cloud. It is efficient because it can be integrated into existing full-precision training pipelines with only minimal code changes.
Its key features are as follows:
- Quantization Support: Supports quantization for weights (including Embedding, lm_head), Activation functions, and KV Cache.
- Mixed-Precision: In addition to 1.58-bit and 4-bit uniform quantization, it supports Matrix-wise mixed-precision such as 2.79-bit and 1.88-bit to optimize performance and efficiency.
- Flexible Distillation: Logits, Features, and Attention distillation can be freely combined and used through a unified interface.
The source code and a collection of quantized models are currently available via GitHub and Hugging Face.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Jevper Released: A Jev Classification Wrapper for OpenAI-Compatible Models with Automatic Logprobs Detection