tandem.cpp v0.3.0 releases with 2-6x faster prompt processing and Project Maya GGUF support

·2026.11.16 21:22

Key point

The update enables 4096-token batches that borrow VRAM during long prompts, achieving up to 1,354 tok/s on Qwen3.8 and 276 tok/s on DeepSeek-V4.1.

Details

  • tandem.cpp v0.3.0 is the first tagged release, featuring an expert cache that processes prompts in 4096-token batches by temporarily borrowing VRAM.
  • Prompt processing speeds improved 2-6x compared to v0.2.0, reaching 1,354 tok/s on Qwen3.8 IQ3_S and 276 tok/s on DeepSeek-V4.1 IQ2_XS with 15k+ token prompts.
  • Decode performance after long prompts reaches 110 tok/s on Qwen3.8 and 16 tok/s on DeepSeek-V4.1, outperforming llama.cpp and Strata in most tested scenarios.
  • Project Maya's DeepSeek-V4.1 Maya-XS GGUFs are supported via a one-time conversion script, with tandem.cpp achieving 1.5x faster decode speeds (59.4 tok/s vs 39.6 tok/s) on an RTX 5090 setup.
  • Current limitations include Linux-only support, disabled VRAM borrowing on AMD ROCm builds, and SSD reads for DeepSeek-V4.1 IQ2_XS on 128 GB RAM systems during long prompts.

This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.

Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.