Building a State-Aware Agent Harness with LangSmith

Most conversational AI evaluates outcomes after the conversation ends, but Candidly developed a state-aware agent harness that identifies user states in real time during conversations and adjusts response strategies. This system analyzes partial conversation traces to infer the user's current state and selects optimal response functions based on transition patterns from past similar conversations.

To predict conversation outcomes, they built a hybrid labeling pipeline. Clear cases (e.g., explicit complaints, no response after the first message) are handled with deterministic rules, while ambiguous cases are routed to an LLM-as-judge evaluator based on conversation patterns. This pipeline achieved a 92.3% agreement rate when compared against a human-labeled LangSmith dataset.

This approach provides a policy surface that allows agents to intervene in real time, rather than relying on simple post-hoc evaluation. This enables Candidly's AI financial planner 'Cait' to more accurately understand user intent and provide effective assistance during high-stakes financial decisions such as debt repayment, savings, and retirement.