ds4.c (GitHub repository)

ds4.c is a small native inference engine built specifically for DeepSeek V4 Flash. Centered on a Metal graph executor, it bundles DS4-specific loading, prompt rendering, KV state, and server API integration together, choosing an execution path tailored to one model rather than being a general-purpose GGUF runner or framework.

The creators note that because active parameters are small, it's fast, and in thinking mode—if you avoid max thinking—the thinking segments are short and proportional to problem complexity. 1M-token context, better English/Italian output, compressed KV cache with disk persistence, and a special 2-bit quantization are presented as strengths. This 2-bit combination is claimed to be stable even for coding agents and tool calling. Since 1M context consumes about 26GB of memory, on a 128GB Mac around 100–300k is described as realistic.

The development direction is to align the engine, specialized GGUF, and agent-based verification together. It's tuned against logits from the official implementation and long-context tests, and the developers state that humans led the ideas, testing, and debugging while GPT 5.5 provided strong assistance. The current code is alpha quality, and it borrows heavily from llama.cpp and GGML in terms of kernels, the GGUF ecosystem, and quantization design. The entire implementation is Metal-only, and the CPU path exists for correctness checking only—but due to a current macOS bug, they warn it carries a risk of kernel crashes.

• Model weights only support the project's own DeepSeek V4 Flash GGUF; arbitrary GGUF files won't work due to mismatched tensor layout, quantization combinations, metadata, and MTP state.
• q2/q4 use a special asymmetric 2-bit quantization that only compresses the routed MoE experts, keeping up/gate at IQ2_XXS and down at Q2_K to preserve quality. q2 targets 128GB RAM-class machines, q4 targets 256GB+ machines.
• MTP, downloaded via ./download_model.sh mtp, is optional speculative decoding, but currently only offers a modest speed improvement.
• The CLI offers a -p one-shot mode and an interactive ds4> mode, with commands like /think, /think-max, /nothink, /ctx N, /read FILE, /quit to control thinking mode and context.
• The server supports OpenAI/Anthropic-compatible APIs (POST /v1/chat/completions, POST /v1/completions, POST /v1/messages) with SSE streaming and tool calling, processing sequentially on a single Metal worker with no batching.
• The agent client keeps context within the server's --ctx limit and allows very long responses with an output limit of 384000 tokens.
• Benchmarks on M3 Max 128GB q2 recorded short 58.52/26.68, long 250.11/21.47, while M3 Max q4 was N/A. On M3 Ultra 512GB, q2 showed 84.43/36.86 and 468.03/27.39, and q4 showed 78.95/35.50 and 448.82/26.62.