spinf-serve: Open-source inference server for bulk LLM scoring on single GPU

Details

spinf-serve is an open-source (Apache 2.0) inference server optimized for bulk classification and decision tasks on a single 24 GB NVIDIA GPU.
It achieves 491 questions/sec and 11,200 tokens/sec on an RTX 4090 with the Gemma 4 12B fp8 model, outperforming vLLM 0.26 which managed 200 questions/sec.
The system uses a "one read, many questions" architecture where content is processed in a single pass, branching to answer multiple fixed-option queries without generating new text.
Supports OpenAI-compatible API, multimodal inputs (text, image, audio, video), and extraction slots for text, numbers, and amounts.
Requires Linux, Python 3.12, and CUDA 13; default model is spinf/gemma-4-12b-fp8 served on port 8000.
Benchmarks show 2.7 seconds to process 1,344 questions on 24 Wikinews articles compared to 6.7 seconds for vLLM.

Key point