27B Optimization for the 3090

Qwen3.6-27B was quantized with 4-bit GPTQ-Pro to make it realistically runnable on consumer-grade Ampere GPUs.

The target cards are ones like RTX 3090, 3080, A5000, A6000, and the goal is to serve a 27B reasoning/coding model locally at speed without datacenter-grade hardware.

The key points are as follows.

• Reduced quality loss via GPTQ-Pro + FOEM quantization
• High-throughput inference support via Marlin optimization
• Tested on 2× RTX 3090
• Generation speed of about 64 tok/s
• First-token response time of about 54 ms TTFT
• Large context support via vLLM
• Apache 2.0 license

The example setup to get started uses vllm serve with options such as gptq_marlin, tensor-parallel-size 2, and max-model-len 132144.

The model link is groxaxo/Qwen3.6-27B-GPTQ-Pro-4Bit, and the project is GPTQ-Pro.