Numan Builds Eval-First AI Health Coach Using LangGraph and LangSmith to Enforce Regulatory Boundaries

Details

Numan developed an eval-first framework for its AI health coach, Nu, to ensure high-quality wellness coaching without crossing into regulated medical diagnosis or prescription.
The team replaced static single-turn fixtures and scripted multi-turn conversations with a layered evaluation system that scores per-category and simulates dynamic patient interactions.
By running evaluations through the same LangGraph production graph, Numan achieved a one-day migration between frontier LLM providers, a process that typically takes weeks of manual testing.
Safety guard nodes and clinician-signed-off rubrics gate every release, ensuring the agent remains a wellness product under MHRA guidance rather than a medical device.
The stack utilizes LangGraph for agent orchestration (supervisor and domain specialists) and LangSmith for tracing, dataset management, and side-by-side provider comparison.
Future plans include online evaluators that continuously score real conversations and automated prompt optimization loops driven by trusted rubrics.

Key point