LangSmith Officially Launches 'Polly,' an AI Assistant Usable Across All Work Environments

Agent debugging is highly complex due to traces spanning hundreds of steps and prompts running thousands of lines. To address this problem, LangSmith has officially launched Polly, an AI assistant that reads traces of 300+ steps and pinpoints the cause of failures.

Previously available only in a limited way on certain pages, Polly can now be used across all pages of LangSmith (tracing, runs, threads, experiments, datasets, annotation queues, evaluators, playground).

Key updates include the following:

• Maintains conversational context: Even when moving between pages, it remembers previous debugging context, minimizing disruption to the workflow.
• Performs actual actions: Beyond just answering questions, it directly updates prompts, creates datasets based on failed run results, writes evaluator code, and compares experiments.

Polly follows the user's workflow to provide various kinds of assistance. In the thread view, it can analyze user sentiment or whether an issue was resolved, and it can directly write or improve evaluator logic. It also compares results from multiple experiments to provide data-driven recommendations on which model or prompt is most effective.

Polly can be instantly invoked with the Cmd+I (Mac) or Ctrl+I (Windows/Linux) shortcut, and can be started right away by adding the API key of the model provider being used as a workspace secret.