AI Briefing
KO

40 tok/s on a 3080

·2026.04.17 05:23

Key point

Using turbo3, achieved about 40 tok/s running Qwen3.6 35B A3B at 260k ctx on a 12GB VRAM 3080.

Details

By combining GPU-POOR compile flags with the turbo3 KV cache, pushed Qwen3.6 35B A3B to about 40 tok/s at 260k ctx on roughly 12GB VRAM with an RTX 3080.

Configuration used:

  • Compile: GGML_CUDA_FA_ALL_QUANTS=ON, GGML_CUDA_F16=ON, GGML_CUDA_FORCE_MMQ=ON
  • Run: --ctx-size 0, --fit on, --flash-attn on, --cache-type-k turbo3, --cache-type-v turbo3, --reasoning off
  • Model: unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M

The author noted applying Qwen3.6's recommended thinking off setting, and structured the prompt harness into ask -> validate -> review -> refine/accept stages.

benchmarkqwenquantizationlocal-llmllama-cppgpu-inferencecontext-window
Read originalShare on Threads

This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.

Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.

Related

RedditMica v0.1 4B Released: Open-Source Decision Model for Agent Loops Trained for Under $30RedditTextCLF Releases Calibration-Free TQ Quantization Method and Open-Source Quant FactoryTLDR AIQwen Intelligence Launches Three Mobile AI Agents and Open Benchmark SuitePyTorchKRPrismML Releases 'Ternary Bonsai 2 27B', a Ternary Quantized Model Maintaining 98.2% of FP16 PerformanceHacker NewsApple Releases LensVLM-9B for Processing Compressed Text ImagesAnthropic NewsAnthropic Releases Claude Opus 5.5 with 40% Cost Reduction and Record Alignment Scores