Human Judgment Needed in Agent Improvement Loops

AI agents work best when they absorb not just documented knowledge but the tacit knowledge hidden within a team. So what matters isn't just picking a model, but designing how to put domain experts' judgment into the improvement loop.

In the trader copilot example, the key is two layers of context. One is financial domain knowledge for interpreting requests like "today's exposure" or "recent volatility," and the other is DB operations knowledge, such as which tables can be trusted and which query patterns are risky. This knowledge needs to be organized with the appropriate stakeholders, such as risk, compliance, and data science.

There are three main points where human judgment enters agent design.

• Workflow Design: Even if the LLM determines the order, regulated or high-risk sections should be enforced in code so that verification is always passed through.
• Tool Design: Strike a balance between flexible but risky tools like execute_sql and safer but more limited parameterized tools.
• Agent Context: Instead of cramming everything into a single system prompt, structure documents, examples, and domain rules so they can be pulled in only when needed.

The core point this piece emphasizes is that having humans design and calibrate automated evaluations creates far higher leverage than manual review that consumes a lot of human time. Using something like LangSmith's Align Evaluator to tune LLM-as-a-judge to match expert judgment, and connecting the results to datasets, evaluations, tracing, and automations, can speed up improvement.

During development, first build a test set from minimal scenarios and expected behaviors, and keep adding cases discovered during actual testing to the dataset. After deployment, use online evaluation and monitoring to detect errors, latency, and negative feedback, and use annotation queues to select only the traces that need expert review. Adding exploratory tools like Insights Agent on top of this can also surface improvement points that are easy to miss with structured evaluation alone.