Microsoft and ThinkingBox-Training Released: Qwen3.8-27B Repeat Success Rate Improved 2.8x

Microsoft and Hugging Face have open-sourced ThinkingBox-Training, which uses DB state determination as reinforcement learning reward
Training Qwen3.8-27B for 50 steps with GRPO increased the number of tasks with 20 consecutive successes from 38 (7.50%) to 107 (21.10%), a 2.8x increase
The cost per task for repeat success decreased by 67.2% from $24.36 to $8.00, but pass@20 dropped from 89.35% to 87.97%
As of October 25, 2026, immediate reproduction is not possible due to inaccessibility of the required data repository

Training Mechanism and Configuration

The GRPO algorithm is used to select only tasks with mixed successes and failures to secure training signals
Offline policy relative difficulty is estimated as a posterior distribution based on a Jeffreys prior to filter difficulty intervals
The environment uses 24 H100s (3 nodes) with the AdamW optimizer and the Verl 0.9.0 framework

After training, Qwen3.8-27B's pass@1 rose to 60.78%, but it did not reach Claude Opus 5.5 (67.16%) or GPT-5.4 (65.36%)
In the repeat success (all 20 successes) metric, GPT-6 Astra (231 tasks) and Claude Opus 5.5 (241 tasks) still hold the advantage
The 20-consecutive-success rate in the retail domain rose the most, from 24.5% to 49.0%

The results are interpreted as a sharpening phenomenon where the probability distribution concentrates on repeatedly passing paths
Benchmark data is restricted to evaluation only, requiring separate task preparation for training reproduction
There are inconsistencies between the paper and the blog regarding evaluation settings (maximum length, parser) and error handling methods