AI BriefingPodium Optimizes Agent Behavior and Reduces Engineering Intervention by 90% with LangSmith
Key point
Podium used LangSmith to optimize AI agent performance and reduce engineering intervention by 90%.
Details
Podium, which helps small businesses with customer service, launched its AI-based agent product, AI Employee. Initially using the LangChain framework, Podium adopted LangSmith as agent use cases became more complex and visibility into LLM calls and interactions became necessary.
Podium focused on building feedback loops throughout the entire agent development lifecycle. Key testing approaches include:
- Baseline dataset curation: Creating an initial dataset representative of basic use cases
- Offline evaluation: Assessing performance against baseline requirements before production deployment
- Feedback collection: Direct user input and real-time online evaluation based on LLMs
- Optimization: Prompt tuning, adjusting search mechanisms, and fine-tuning models based on trace data
- Continuous evaluation: Offline evaluation and dataset expansion through backtesting and pairwise comparison
Through these processes, Podium was able to provide high-quality customer support on its AI platform without engineering intervention, ultimately reducing engineering intervention by 90%.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.