A Taxonomy of RL Environments for LLM Agents

While model architecture and post-training recipes get a lot of attention, the core factor that actually determines what an agent learns is the RL (Reinforcement Learning) environment. The environment in which a model practices, how it is evaluated, and what tools it uses all shape the agent's practical capabilities.

An RL environment for LLM agents can be defined as a set of five core components: $E = {T, H, V, S, C}$.

• $T$ (Tasks): the set of problems the agent must solve
• $H$ (Harness): the interface that drives the model
• $V$ (Verifier): the verifier and reward function
• $S$ (State management): state management
• $C$ (Configuration): environment configuration

In particular, the design of Tasks is critically important. Tasks vary structurally, from simple Q&A to multi-hop retrieval, code generation, and enterprise workflows, and the required capabilities differ depending on the number of actions the agent must perform and the types of tools involved. Failing to design an appropriate Task Distribution can lead to problems such as the agent gaming the reward function or failing to handle the uncertainty of real production environments.

The sequence of states, actions, and rewards that an agent generates while performing a task is called a Trajectory, and a single run-through of this process is called an Episode. Additionally, the execution log containing tool calls and observation results is referred to as a Trace.