Challenges in On-Policy RL for Agentic Workflows: Harness Consistency and Tree Training

Reinforcement learning for agentic use-cases relies on the on-policy assumption, where the model generating rollouts and the model being updated are identical. This is critical for algorithms using PPO's importance ratio, which should ideally be 1.0. However, as sessions lengthen—Alibaba's QwenGyre paper reports rollouts on a 122B model averaging 2 hours and the 2.4T flagship exceeding 4 hours—maintaining this consistency becomes difficult due to external factors like harnesses and sandboxes.

Harness and Environment Consistency

Agentic post-training involves more than just the model; it includes inference engines, harnesses, and sandbox environments. Alibaba trains within proprietary harnesses like Claude Code to ensure relevance but faces challenges with black-box control flows. To manage GPU utilization, they use a proxy to reroute requests and shuttle KV cache between training and rollout cells. DeepSeek uses a custom harness, moving it inside the sandbox in version 4.1 to freeze state (DRAM and file system) during preemption, avoiding the need to reconstruct agent loops.

Trajectory and Tree Complexity

Tool calls introduce extra tokens that must be masked to prevent training on non-model outputs. Inference servers parsing text can alter values (e.g., false to False), requiring solutions like Prime Intellect's Renderers library or QwenGyre's proxy caching of original payloads. Modern agents spawn subagents and make parallel calls, creating tree-structured trajectories. Tree Training serializes these trees into streams, weighting tokens by branch frequency. QwenGyre caps trainable paths at 5 and averages weights to prevent subagent-heavy rollouts from dominating gradients.

Training-Inference Mismatch

Inference engines (e.g., SGLang, vLLM) prioritize latency with small batches, while training engines (e.g., Megatron, TorchTitan) prioritize throughput with large batches. This leads to different kernels and bitwise differences in float operations. Solutions include recomputing old log-probs (costly) or clipping ratios (a fudge criticized by Megatron developers). Achieving bitwise equivalence is difficult; fixing reduction ordering can cost 10-20% performance. Meta and Colfax research highlight these costs, while AReaL-TIK is an agentic loop that starts from a single bitwise-consistent kernel shared by inference and training, using agents to optimize speed while maintaining parity.