Qwen3.5 model quantization: why do community versions underperform

Unsloth analyzed the cause of performance degradation in community-released Qwen3.5 MLX quantized models through 150+ benchmarks and 121 configuration comparisons.

The core problem is that most community tools use uniform quantization, applying the same bit count to every layer. Qwen3.5 has a hybrid structure where standard self-attention alternates with GatedDeltaNet-based linear attention, and in this structure, the sensitivity to information loss varies dramatically across specific layers.

In particular, the linear_attn.out_proj layer was highly vulnerable to 4-bit compression. This layer's sensitivity to information loss was measured at approximately 120x higher than lm_head, resulting in tool calling errors, meaningless outputs, and hallucination.

Unsloth's solution is mixed-bit quantization, which assigns different bit counts per layer.

• Low-sensitivity MLP layers get 3-bit
• attention Q/K/V layers get 5-bit + AWQ applied
• The most sensitive output layer stays at bf16

Calibration data also mattered. Instead of generic text like Wikipedia, using conversation, coding, and tool-call examples enabled importance calculations that matched real-world usage.

The results are clear. tool calling, structured output, and code generation quality improved significantly over existing community versions, and MLX achieved performance on par with the equivalent GGUF version. However, since some sensitive layers must remain in bf16, disk size is larger than pure low-bit models.

The key takeaway is that compression shouldn't simply mean reducing bit counts, but rather requires understanding the model's internal structure and layer sensitivity. In practice, when using lightweight models, it's essential to check the deployer's quantization method and calibration data.