All You Need Is an RL Environment

RL environments represent a new form of data required for AI agents. While traditional deep learning trained and tested models using curated data, agents develop capabilities by acting within environments and evaluating outcomes. The term 'RL' is merely conventional; reinforcement learning is not strictly required.

The advancement of deep learning stemmed from moving away from manually writing rules and heuristics, instead scaling data and compute to let models learn problems. To apply the same principle to agents, we need to define what an agent is and find ways to update each component using compute.

An agent is fundamentally a combination of an LLM, a system prompt, and a harness responsible for tool use and execution. Each of these three elements can be improved using compute and environments.

• LLM: Updates weights using trajectories generated in the environment via RL, SFT, and midtraining.
• System Prompt: Explores for the best-performing prompts using methods like GEPA, autoresearch, and evolutionary algorithms.
• Harness: Currently improved mainly through manual effort, but could evolve in the long term toward automatically generating and optimizing the software itself.

Optimizing prompts or harnesses requires environments and scoring functions that systematically measure agent performance. Additionally, like model training, we must establish train/test splits to ensure optimization does not overfit to specific data and generalizes to new tasks.

Ultimately, the most critical foundation in agent development is building good RL environments. Only with proper environments can compute be invested iteratively to improve model weights, system prompts, and execution harnesses.