Why LLM Serving Needs to Separate CPU and GPU

Shepherd Model Gateway(SMG) proves the claim that LLM serving needs to separate CPU and GPU with an actual product. When tokenizing and detokenizing became a bottleneck blocked by the Python GIL in SGLang and vLLM, SMG moved all of it to a Rust gateway.

The core architecture is Clients → Gateway → Router → Workers. The gateway handles tokenizing, reasoning and tool call parsing, multimodal preprocessing, MCP tool orchestration, chat history management, structured output validation, and stop sequence detection, while the GPU engine only receives preprocessed tokens and tensors.

Technically, the native Rust gRPC data plane is central. The tokenizer operates in Rust with a two-tier cache (L0 exact-match, L1 prefix-aware), and reasoning parsing happens in real time during streaming. Multimodal preprocessing reimplements the Hugging Face image processor from Python into Rust to support major vision-language models such as Llama 4 Vision and Qwen VL, and MCP and WASM middleware are also handled independently in the gateway.

SMG offers a broad set of features today.

• Supports 5 native Agentic APIs: Chat Completions, Responses API, Messages API, Interactions API, Realtime API
• Integrates with SGLang, vLLM, TensorRT-LLM, MLX and external model providers
• Cache-aware routing restructuring achieves 10-12x faster insertion speed and 99% memory savings
• Verified with 1,082 comparison points across 8 H100 models, 2 runtimes, 5 traffic scenarios, and 9 concurrency levels

In benchmarks, gRPC showed about 8% higher throughput than HTTP at concurrency 256, and in a prefill-decode disaggregated environment, TTFT average decreased by 23% and p99 decreased by 28%. The conclusion is clear: to keep the GPU as busy as possible, the CPU work that used to sit right next to the GPU needs to be boldly separated out and operated as its own serving layer.