Building AI Agents from First Principles

Most post-training tutorials start with installing a framework. But to understand the whole system, you need to start at a lower level. Before there's a trainer, there's an environment; before there's reinforcement learning, there's an action space; and before there's an agent, there's a policy that generates actions which change the state of the world.

The core loop is this: prompt → model action → environment → reward → gradient update. Browser agents, coding agents, robotics planners, and diagram agents all share this same structure. The difference lies in the environment, the action space, and the reward function.

This post uses a text-to-diagram agent as its example. The model outputs structured JSON actions to create shapes on a canvas. For instance, it uses create_shape to create a rectangle and connect to link an arrow. The model isn't simply generating text — it's generating commands for another system to execute.

Environment implementation presents a minimal canvas environment written in pure Python. It supports rectangles, ellipses, diamonds, text blocks, and arrows, providing a deterministic world where model outputs can succeed or fail. It validates constraints such as preventing duplicate IDs, bounded coordinates, and valid shape types.

Reward function design is the hardest part. A trainer can only optimize the reward it's given. If the reward is mostly syntactic, the model learns syntax; if the reward measures task satisfaction, the model has a chance to learn useful behavior. The example reward function combines:

• Parseability (0.4)
• Layout quality — avoiding overlap, presence of labels, arrow connectivity (0.3)
• Semantic coverage — inclusion of important words from the user's request (0.3)

Why is SFT needed before RL? If you start RL from a model that can't produce valid actions, nearly every rollout gets a reward of 0. This isn't an optimization problem — it's a state distribution problem. The model must first learn the language of the environment before it can optimize the reward.

Frameworks like TRL, Unsloth, PRIME-RL, and verl aren't magic. They're just infrastructure (batching, rollout generation, distributed inference, reward computation) built around this loop. The conceptual core is much smaller.