LangSmith Used as Clinical Evaluation Infrastructure to Ensure Reliability in Medical AI

Infrastructureizing Clinical Expert Judgment

The core bottleneck in medical AI development is the scarce validation time of clinical experts. Abridge and Included Health utilized LangSmith to convert these expert judgments into reusable infrastructure such as labels, judges, and datasets. The goal is to progressively compound reliability and reduce dependency on experts outside the engineering team.

Refining Evaluation Systems: Dual Judge Strategy

Clinical evaluation is complex because there is no single correct answer, and it must consider 'Correct inaction.' To address this, Abridge employs two types of judges in parallel.

• Reference-free Judge: Measures generalized accuracy during development and operations by comparing directly with the original conversation.
• Reference-based Judge: Captures the context and nuances of specific specialists by comparing with curated examples.

Additionally, known failure modes are categorized by frequency and severity, and dedicated judges for each item are generated using an automatic prompt optimization framework to ensure scalability.

Shortening Release Cycles and Production Reliability

By linking evaluation results to release gates, deployment speed and safety were improved simultaneously.

• Abridge: Automated the process from offline evaluation to limited A/B testing (silent rollout to 10~15% of customers), shortening the release cycle from the previous 1~2 months to just a few days.
• Included Health: Completed the migration of the AI guide 'Dot' regression-free within 2 weeks, achieving a 75% increase in chat engagement and over 99% high-risk situation identification rate post-launch.

PHI Management and Security Architecture

Given the nature of medical data, PHI (Protected Health Information) management is essential from the design stage of the evaluation infrastructure. Abridge mandates self-hosting, access control, and auditability as essential requirements, adopting a control method that removes identifying information at the data ingestion stage. LangSmith offers various deployment options from managed cloud to self-hosted, enabling teams to directly determine data storage locations and audit controls.