Customizing multi-turn AI agents with reinforcement learning

AWS AI Labs validated reinforcement learning (RL)-based customization to quickly optimize general-purpose AI agents for specific work environments. Two experiments were conducted: one on a personal assistant-type agent, and the other on agentic RAG.

The core premise was asynchronous multi-turn agents and verifiable ground-truth signals. Rather than relying on human simulation, the environment's own outcomes were used directly as rewards for training. Public benchmarks and environment simulators such as AppWorld were used, allowing the focus to stay on the RL methodology itself.

The training pipeline consisted of two stages.

• An online simulator receives a batch of tasks and generates rollout trajectories, computing a reward for each trajectory via ground-truth verification.
• An online RL trainer takes these trajectories and rewards, updates the actor policy, and syncs the updated weights back to the agent in the simulator.

In the AppWorld experiment, an agent operating across the file system and multiple apps broke down an instruction such as "add a date prefix to files and move files not from the current year to trash" into 32 API calls. It demonstrated complex multi-turn behavior, including error handling—such as switching to move_file when rename_file wasn't available—date/time parsing, and maintaining state across directories. The reward was a sparse reward collected only at the final turn, but this approach actually proved advantageous for performance improvement.

The results were clear: RL substantially boosted performance even with small amounts of data.

• AppWorld / Qwen2.5-32B-Instruct: base model 39.20% → 72% after RL
• For comparison, Sonnet 3.7/4.0 scored around 69%
• NQ / Qwen2.5-3b-Base: 0.106 → 0.406
• Musique / Llama-3.2-3B-inst: 0.04 → 0.1

An important pattern observed here is that larger base models show greater absolute improvement from RL. Better rollouts lead to better training signals, which in turn create a virtuous cycle where RL works even better. It was also noted that applying online RL to stronger base models could potentially surpass some current proprietary benchmarks.

The cost implications were also significant. With just 72 AppWorld examples, performance approached near-proprietary levels, while reducing costs to roughly 1%–2%. In some cases, effects appeared from the very first training step, reaching competitive performance within 30 steps.

At the same time, RL didn't just produce simple score increases. Models learned the habit of checking API documentation before writing code, reducing code errors, and maintained semantic understanding even when exact match scores dropped. On the other hand, smaller models showed fundamental reasoning limitations in determining "unanswerable questions" or extracting correct answers from relevant context—in such cases, RL alone was insufficient, and distillation could be more effective.

The practical recommendation is clear: online RL is worth investing in for customizing various agents, including assistant agents and coding agents. However, the conditions for success are also clear: data quality and format accuracy matter, stronger base models yield greater gains, and during training, it's more efficient to prioritize harder tasks over easier ones.

Future research follows two tracks. One is improving training efficiency through synthetic data generation and adaptive data filtering. The other is digging deeper into the RL algorithm itself—comparing across model families, exploring signals beyond outcome-based rewards, and optimizing the pipeline. The co-mentioned works SALT: Step-level advantage assignment for long-horizon agents via trajectory graph and Reinforcement learning for self-improving agent with skill library show further progress in this direction.