Tencent Compresses Open Source Hy4 Preview 770B Model to 214 GiB Using Sherry Quantization

Tencent’s Hunyuan quantization team developed Sherry, a new algorithm that compresses the Hy4 Preview 770B-parameter model from approximately 1.5TB to 214 GiB. The compression retains all 770B parameters by altering weight representation rather than pruning, utilizing a new storage format called STQ1_0 and a mixed-precision allocation strategy named MIX-STQ1_0.

Quantization Technique

The core innovation encodes four weights into five bits using a ternary pattern where each group of four weights takes values from {-d, 0, +d} with exactly one zero. This structure yields 32 patterns (4 zero positions × 8 sign combinations), resulting in 1.25 bits per weight. With a shared FP16 scale every 256 weights, the STQ1_0 format averages 1.3125 bits per weight, while the full mixed-precision model averages about 2.38 bits per weight.

Sensitivity and Precision Allocation

Precision is allocated based on weight sensitivity measured via Full-Hessian scoring ($H = XX^T$), which captures correlations missed by diagonal-only imatrix scoring. The Hy4 Preview model features 77 MoE layers with 256 routed experts each. Expert weights undergo the hardest compression, while sensitive components are protected:

• Expert gate/up projections: MIX-STQ1_0 applies IQ2_XXS to 48 sensitive layers and STQ1_0 to 29 less sensitive layers.
• Optimization: This selective approach produces less error than using the intermediate IQ1_M format everywhere at the same average bit budget.

Performance and Runtime

The quantization is post-training, requiring no retraining. The encoder alternates between fitting the scale $d$ with weighted least squares and choosing the zero position via imatrix-weighted error. Across 1,200 rows of real expert weights, three alternating rounds reduced weighted reconstruction error by roughly 90% compared to the original ternary encoder.

• Inference Speed: STQ1_0 CUDA kernels added to a patched llama.cpp build run roughly as fast as IQ1_M despite the lower bit width.
• Accuracy: The model shows nearly unchanged MRCR retrieval performance and a small drop in math tasks. Against UD-IQ1_M at a similar bit budget, MIX-STQ1_0 led across reported evals, including a gain of more than five points on MRCR.