Mercor and SkyRL Release RL Training Guide for 397B-Parameter Knowledge Work Agent

Mercor and SkyRL have released a reinforcement learning (RL) training guide for the Qwen3.5-397B-A17B model's knowledge work agent. The guide includes the full training scripts, model weights, and evaluation traces, reproducing a 70% improvement in Pass@1 scores on the APEX-Agents benchmark, from 16.11% to 27.29%.

Training Infrastructure and Risk Mitigation

The core of the training is reducing failed trajectories that waste GPU time and ensuring accurate token accounting (TITO). Using the Harbor framework, agents are executed in containerized environments, performing tasks in simulated company environments containing dozens of files and servers via MCP servers and code execution tools. To minimize environment errors, timeouts were set for all interactions, and strategies such as bypassing LLM judge rate limits were applied.

Algorithm Optimization and Hero Run

Through ablations on the 35B model, it was confirmed that the DPPO loss function, prompt_mean aggregation method, and harness nudge (which encourages completion when 20% of the context budget remains) were the most effective. Applying this configuration to the 397B model in a 'Hero Run' showed the largest performance improvement in the management consulting field compared to the 35B model. Performance improvements largely transferred to other code-based harnesses (OpenCode) as well as the MCP-based harness (Archipelago) used for training, proving that post-trained models are reusable assets not fixed to specific scaffolds.