Hugging Face Adds GGUF Model Support to Transformers

Support for running GGUF models from llama.cpp has been added to Hugging Face Transformers. By reusing the ggml kernel as the kernels library, local inference is now possible using only the from_pretrained API.

Key Features and Support Scope

• Initial Support: Limited to Apple Silicon (MPS) and the Qwen3.5 architecture (dense/MoE)
• Quantization Support: Can load various GGUF quantization files ranging from BF16 to Q4_K_M
• API Compatibility: Standard Transformers APIs (such as generate) are available after loading
• Server Mode: Can expose an OpenAI-compatible API via transformers serve

Performance and Limitations

• Performance Comparison: On a MacBook Pro M2 Max, Transformers achieved token generation speeds (tg128) similar to llama.cpp
• Current Limitations: Packed inference is exclusive to Apple Silicon, Padding/Batching are not supported, and architecture coverage is limited to Qwen3.5/3.8
• Use Cases: Useful for evaluating GGUF models in Python/PyTorch environments, Fine-tuning (after dequantization), and custom decoding experiments