Hugging Face Introduces Native GGUF Support in Transformers

Hugging Face has introduced native support for the GGUF format in the transformers library. This allows users to directly load llama.cpp-based quantized models by specifying the gguf_file parameter in AutoModelForCausalLM, while utilizing the existing PyTorch-based Transformers API as is.

Key Objectives and Benefits

• Debugging and Flexibility: Debugging models, evaluation, and implementing custom generation logic become much easier by using tools from the PyTorch ecosystem as is.
• Hardware Compatibility: In Apple Silicon (e.g., M2 Max) environments, ggml kernels can be reused to run quantized weights as is.
• Performance Comparison: Test results showed that Transformers achieved inference speeds similar to llama.cpp. Based on Qwen3.5-4B (Q4_K_M), it recorded 70.4 tokens per second (llama.cpp: 71.8 tokens), and based on Qwen3.8-27B (UD-Q4_K_M), it recorded 15.9 tokens per second (llama.cpp: 13.4 tokens).

Purpose of Use

This feature is not intended to replace llama.cpp for maximum local inference performance. Instead, it focuses on enabling the use of the same GGUF models in a more flexible and developer-friendly PyTorch environment.