AI Briefing
KO

vLLM Benchmark on 4x 5060 Ti

·2026.04.30 01:59

Key point

On 4x 5060 Ti, vLLM recorded pp 1444.9 tok/s and tg 47.4 tok/s.

Details

To handle an extremely long prompt for a browser car racing game, I ran vLLM on 4x 5060 Ti and measured pp and tg throughput together.

  • Model: Qwen/Qwen3.6-27B-FP8
  • Config: tensor-parallel-size 4, max-model-len 262144, qwen3_next_mtp in speculative-config, num_speculative_tokens=2
  • Additional options: fp8 quantization, max-num-seqs 2, enable-prefix-caching, enable-auto-tool-choice, qwen3_coder parser, language-model-only
  • Installation: uv pip install vllm --torch-backend=auto --extra-index-url https://wheels.vllm.ai/nightly

Measurements

  • >10k token prompt processing (pp): 1,444.9 tok/s
  • Generation (tg): 47.4 tok/s
  • Draft acceptance rate in logs: 70.4% and 97.6%

The service was registered with systemd, and logs were checked with journalctl -f -u vllm.service.

benchmarkvllmllm-inference5060-ti
Read originalShare on Threads

This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.

Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.

Related

PyTorchKRvLLM Introduces Hardware-Agnostic Layer Architecture to Support Diverse AcceleratorsPyTorchKRvLLM rope_scaling Override Causes 36% Model Configuration MismatchRedditTextCLF Releases Calibration-Free TQ Quantization Method and Open-Source Quant FactoryRedditComparison of Features and Use Cases for 8 LLM Inference StacksApple MLApple Releases CapQuiz, a Reference-Free Benchmark for Video Caption Quality AssessmentHacker NewsvLLM Publishes Performance Benchmarks for 5 Speculative Decoding Methods on AMD GPUs