monday Service + LangSmith: A Code-Centric Evaluation Strategy Built from Day One

monday Service is an AI-native Enterprise Service Management (ESM) platform that included Evaluation as a mandatory requirement from the early stages of development when building an AI Service Workforce to automate customer interactions.

They utilize LangGraph-based ReAct agents, and adopted a two-tier evaluation structure to prevent errors that arise from the complex reasoning steps characteristic of agents.

1. Offline Evaluations (Safety Net): Operating like unit tests, this verifies core logic (Groundedness, Retrieval Accuracy, Tool-calling) and edge cases through a curated Golden Dataset. This prevents prompt modifications from breaking existing functionality.

2. Online Evaluations (Monitor): This continuously monitors the agent's performance in the actual production environment. It tracks and improves business metrics such as Automated Resolution rate and Containment rate in real time.

Through this Evals-driven development framework, they shortened the evaluation feedback loop by 8.7x (162 seconds → 18 seconds), and built an environment where hundreds of cases can be comprehensively tested in just a few minutes.