antirez/ds4 - A Local Inference Engine for DeepSeek V4 Flash on Metal

antirez has released ds4.c, a local inference engine built exclusively for DeepSeek V4 Flash. Rather than a general-purpose GGUF runner, it's designed as a native C implementation tailored to Apple Metal GPU, with its core path specialized for DeepSeek V4 Flash.

Key features include the following.

• Built-in OpenAI / Anthropic-compatible HTTP API server, allowing direct integration with coding agents like Claude Code, opencode, and Pi.
• Support for a 1 million token context window with a heavily compressed KV cache, plus disk-based KV cache persistence.
• Support for thinking, non-thinking, and Think Max modes, with tool calls converted between DeepSeek's DSML format and Anthropic tool_use blocks.
• Built on top of llama.cpp and the GGML ecosystem, but the executor is Metal-only, with the CPU path provided only at a level sufficient for correctness verification.

Only the DeepSeek V4 Flash GGUF weights published for this project are compatible. The 2-bit quantization asymmetrically quantizes only the MoE experts, and it's designed to run even on Macs with around 128GB of memory. Download scripts for q2, q4, and an optional MTP (speculative decoding) model are also provided.

Benchmarks were also presented. Prefill and decode throughput for the q2/q4 variants was measured on an M3 Max 128GB and an M3 Ultra 512GB, showing high prefill speed even with long prompts. However, the server currently serializes inference through a single Metal worker, so it does not handle concurrent requests in parallel.