Introducing LangSmith Engine

Existing AI agent development required a manual, repetitive process of directly reading traces, identifying failure patterns, fixing prompts or logic, and rebuilding datasets. In this process, figuring out what to fix or preventing error recurrence at scale was extremely cumbersome and difficult.

LangSmith Engine automates this entire development cycle. It monitors production traces to cluster failure cases into named issues, and analyzes connected code repositories to diagnose root causes. It then automatically drafts a PR (Pull Request) to resolve the problem and a customized online Evaluator.

Key features are as follows:

• Issue Clustering and Diagnosis: Identifies failure patterns and analyzes root causes based on the code, providing a prioritized list.
• Automated Fix and Evaluation Suggestions: Proposes code fixes to resolve issues along with customized evaluators to continuously track the issue.
• Evaluation Dataset Enrichment: Automatically incorporates failed traces into offline evaluation datasets to prevent the same problem from recurring.

LangSmith Engine integrates immediately with existing LangSmith setups and infrastructure, so it can be used right away without any separate setup, and is currently available in public beta.