AI BriefingAgentic RL: The Right Way to Implement Token-In, Token-Out
Key point
When training agentic RL, you must avoid token mismatch issues that arise from re-encoding decoded tokens.
Details
Adding tool call capability to an RL (Reinforcement Learning) loop to train an LLM as an agent can cause unstable training curves or errors. This happens because the Token-In, Token-Out (TITO) invariant is broken during the process of parsing the model's responses and re-tokenizing the conversation content.
If you decode text and then re-encode it, the result can differ from the original token IDs. In this case, the gradient is computed on a sequence the model didn't actually sample, delivering a mathematically incorrect signal and breaking training.
Two strategies to solve this are as follows:
- Using a model-specific Renderer: Build a dedicated interface for each model family that handles message formatting and the connections between turns. Implementation is tricky but accurate.
- The no-re-encoding principle: Follow the rule of "never re-encode decoded tokens." Designing chat templates to have a prefix-preserving property for tool messages helps avoid complex edge cases.
Ultimately, the core of RL is to optimize only on the exact tokens the model generated.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.