Building an AI Health Coach: Evaluation, Safety, and Regulatory Compliance

UK-based digital health company Numan operates Nu, an AI coach that provides nutrition guidance and behavior-change coaching. Nu is designed not to make medical diagnoses or prescriptions, and staying within regulatory boundaries so as not to fall under the 'medical device' category is a key challenge.

Existing approaches such as Single-turn eval using fixed datasets or Scripted multi-turn using pre-written scripts had the limitation of failing to reflect the context of coaching. This is because coaching is not about short-answer responses but about identifying a patient's issues and maintaining appropriate boundaries within the flow of conversation.

To address this, Numan introduced an Agent evals approach using LLM personas. By inputting a patient persona, scenario, key issues to be extracted, and conversation turn limits, a low-cost model plays the patient role and carries on a multi-turn conversation with Nu. This provided the following benefits:

• Broad coverage of the language space: Diverse expressions can be handled through generated personas rather than fixed phrases
• Adversarial probes: Automatically tests questions with potential regulatory violations, such as requests for diagnosis or questions about medication use
• Safe-completion: Measures the ability to appropriately decline and guide questions that fall outside the regulatory scope while maintaining a coaching tone

This system operates through a 4-stage improvement loop. Humans review actual conversations and annotate errors, and an automated script classifies and prioritizes these into a backlog, achieving continuous model improvement.