AI Briefing
KO

22 tok/s on RTX 5060 Ti 16GB

·2026.04.24 09:38

Key point

Qwen3.6 27B IQ4_XS recorded 22 tok/s on an RTX 5060 Ti 16GB.

Details

  • Ran Qwen3.6-27B-IQ4_XS.gguf from unsloth/Qwen3.6-27B-GGUF via llama-server.

  • The settings were -ngl 999 -ctk q4_0 -ctv q4_0 -b 128 -ub 128 -c 24000.

  • Recorded 22 tok/s on an RTX 5060 Ti 16GB.

  • Raising the KV quant further made it hard to exceed 8192 ctx, while raising -b and -ub to 256 allowed up to 16384 ctx.

  • Turning off GNOME freed up an additional 300 MiB, pushing it up to 24k ctx, and 63/65 layers were loaded onto the GPU without quant context.

qwenquantizationllama.cppgpu-inference
Read originalShare on Threads

This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.

Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.

Related

RedditTextCLF Releases Calibration-Free TQ Quantization Method and Open-Source Quant FactoryRedditUkisAI Releases Swift Series: Efficient Qwen-Based LLMs with Reduced Thinking TokensPyTorchKRPrismML Releases 'Ternary Bonsai 2 27B', a Ternary Quantized Model Maintaining 98.2% of FP16 PerformanceHacker NewsApple Releases LensVLM-9B for Processing Compressed Text ImagesTLDR AIQwen Releases 'RecreationWorld' for Training AI Agents to Rebuild AppsTLDR AIQwen Intelligence Launches Three Mobile AI Agents and Open Benchmark Suite