AI BriefingGemma 4 31B MLX Mixed Quantization
Key point
An MLX conversion of Gemma 4 31B mixed-quantized at 3 and 5 bits has been released.
Details
leonsarmiento/gemma-4-31B-it-3bit-mlx has been registered on Hugging Face. It is an MLX version of google/gemma-4-31B-it converted using mlx-lm 0.31.2, aimed at Mac environments with limited RAM.
- Quantization method: only
embed_tokensat 5-bit, all other quantizable layers at 3-bit - Group size: 64
- Total output size: approximately 13.8GB
- Method: used a custom
quant_predicatewithmlx_lm.convert
Recommended sampling settings were also provided.
- temperature 1.0
- top_p 0.95
- top_k 64
- min_p 0.05
- repeat_penalty 1.05
To parse reasoning output in LM Studio, set the start string to <|channel>thought and the end string to <channel|>, and add enable_thinking = true to the ninja template. The usage example is to install with pip install mlx-lm and then call load() and generate().
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Underdog Releases 'Husky', a Model-Specific Inference Engine Up to 4.5x Faster than MLX
Optimizing Gemma 4 Serving with vLLM on Amazon EKS [Part 2: Throughput and Latency SLOs on a Single GPU]Optimizing Gemma 4 Serving with vLLM on Amazon EKS: Cold Start Reduced from 428s to 226s