Paddock Inference Engine Released as Open Source

The inference engine Paddock, developed in Rust and C++, has been released as open source under the MIT or Apache-2.0 license. The engine includes its own CUDA kernels and is provided as a single binary supporting OpenAI and Anthropic-style APIs.

Performance and Compatibility

• Performance: Based on the Qwen3.8-27B FP8 model, it achieved processing speeds 1.02x to 1.19x faster than vLLM across all 13 test benchmarks. It demonstrated up to 37x performance advantage over llama.cpp Q8_0. With 32 concurrent clients, it processed 1062 tokens per second, surpassing vLLM (958 tokens) and SGLang (844 tokens).
• Hardware Support: Verified on Blackwell (5090, RTX PRO series, B200) and Ampere (A6000, 30 series) architectures. Hopper and A100 kernels are included in the code but have not yet been verified on hardware.
• Limitations: Currently CUDA-only, running exclusively on Windows and Linux; Mac, ROCm, and Vulkan are not supported. Additionally, only a single model can be loaded per GPU, making tensor parallelism impossible.