LangSmith Team Answers Common Questions on Agent Evals, Highlighting Cost Reductions and Framework Agnosticism

The LangSmith team released a comprehensive guide addressing common challenges in evaluating AI agents, emphasizing the distinction between Offline Evals for pre-deployment testing and Online Evals for real-time production monitoring. A key focus is cost management for online evaluations, where using Jev (from TypeSafe AI) instead of Claude Sonnet 4.6 reduced total costs from $28.17 to $0.34 while maintaining identical human label agreement rates.

Evaluation Strategies and Evaluator Types

The article outlines four primary evaluator types: Code evaluators for deterministic checks, LLM-as-a-judge for subjective quality, Human review for high-stakes validation, and Pairwise evaluators for comparing outputs. To improve LLM judge reliability, the team recommends using narrow questions and binary pass/fail scales rather than broad 1-10 scales. Offline Evals allow for regression testing against fixed datasets, while Online Evals use sampling and filtering to monitor live traffic without excessive expense.

Framework Agnosticism and Migration

LangSmith supports agents built with any framework, including LangChain, LangGraph, Deep Agents, or no framework at all. Tracing can be implemented via framework integrations (e.g., OpenAI Agents SDK, Claude Agent SDK), model provider wrappers, or the @traceable decorator. Existing evaluation assets, such as datasets and code evaluators, can be migrated to LangSmith with minimal modification, and the open-source agentevals package provides prebuilt evaluators for messages and tool calls.

Benchmarking and Infrastructure

For complex multi-step tasks, benchmarks like Terminal-Bench 2.0 and IssueBench utilize isolated container environments to test agent behavior and final state changes. The Harbor open-source eval harness facilitates these containerized tests, logging each run as a LangSmith experiment. This infrastructure supports evaluating not just final outputs but also the agent's path and environmental side effects, such as file edits or database updates.