SetFit Inference Accelerated 7.8x on Intel CPUs

SetFit is an efficient framework for fine-tuning Sentence Transformers models with a small amount of labeled data. It delivers high accuracy without prompts and offers much faster training and inference speed compared to LLMs.

Using the 🤗 Optimum Intel library, you can maximize SetFit model inference performance on Intel Xeon CPU environments. In particular, applying Post-Training Quantization (PTQ) techniques leveraging Intel Neural Compressor (INC) can significantly improve memory footprint and latency while maintaining model accuracy.

Key results are as follows:

• 7.8x speedup: Through optimization, SetFit inference speed on Intel CPUs can be increased by up to 7.8x.
• Leveraging hardware acceleration: Utilizes the latest Intel CPU acceleration features such as AVX-512, VNNI, and Intel AMX to improve computational efficiency.
• Optimized for production deployment: Quantization enables stable, production-grade service deployment based on Intel Xeon CPUs.