T4 Qwen2.5 Inference Challenge

A public inference challenge has opened, running Qwen2.5-0.5B on a Tesla T4 16GB with 50 concurrent requests.

The harness is fixed, and participants are free to optimize the following:

• quantization
• flash attention
• CUDA graphs
• KV cache strategy
• speculative decoding
• custom kernels
• batching

Scoring is based on 5 metrics, with throughput accounting for 40%.

There are three disqualification conditions:

• TPOT under 50ms
• TTFT under 2000ms
• success rate 100%

As a baseline, a naive vLLM setup achieves about 1,600 tok/s, and the current top score is 3,536 tok/s. The author noted that there is still significant room for optimization in the combination of T4 and small models.

Entry is via submitting a free Colab notebook, and H2LooP will provide top performers with a PS5, a Claude Code subscription, and verified high scorers a technical interview.