Why We Write Our Own C and C++ Inference Engines

LocalAI stated that while most backends wrap existing engines such as llama.cpp, vLLM, whisper.cpp, and MLX, 18 backends were ported directly to C/C++. The background for developing its own engines includes the large installation footprint of Python/PyTorch-based stacks, CUDA/glibc dependencies, and difficulty deploying models that lack a C++ implementation.

The flagship example, vllm.cpp, implements vLLM V1's serving architecture in C++20. It supports paged KV cache, continuous batching, prefix caching, a scheduler, and a sampler, and doesn't even use ggml during inference.

• vLLM installation size: 9.1GiB virtualenv
• vllm.cpp binary: 66MiB
• On NVIDIA GB10, with Qwen3.6-27B NVFP4, throughput across concurrency 1–32 is virtually tied with vLLM
• At concurrency 1, vllm.cpp recorded 86.05 tok/s, 4.5% faster
• Maximum host memory was 24.88GiB, lower than vLLM's 28.18GiB
• Token-level output was identical to vLLM across all test ranges

In a CPU environment using the same GGUF files, prefill was 1.18x faster than llama.cpp, and on Apple M4, warm total throughput was 97.6% compared to MLX-LM.

depth-anything.cpp is an example that ports ByteDance's Depth Anything 3 to C++/ggml. Compared to PyTorch on a Ryzen 9 9950X3D, the q8_0 model showed the following figures.

• Model size: 516MB → 142MB
• Loading time: 749ms → 40ms
• Inference time: 416.9ms → 319.4ms, a 1.31x improvement
• Peak RAM: 1,328MB → 363MB
• The q4_k build is 99MB and produces results nearly identical to the baseline implementation

The key to the performance improvement wasn't the matrix multiplication kernel itself, but rather caching the positional embedding and bicubic embedding that were being recalculated every time, eliminating roughly 95ms of CPU overhead that occurred on every inference. LocalAI explained that its own C/C++ engines provide small deployment binaries, predictable memory usage, and low dependencies, as well as per-model execution path optimization.