Building Clinical AI Agents with LangGraph: Abridge's Eval Stack for High-Stakes Healthcare

Janie Lee, VP of Product at Abridge, walks through how a clinical AI company ships fast without compromising patient safety across 250 health systems. She covers two case studies: their core ambient note product — where misattributed symptoms or wrong dosages carry real legal and clinical consequences — and their Abridge Assistant, a unified agent that persists across the entire patient visit workflow. The talk is a detailed look at how Abridge uses LangGraph and LangSmith to run reference-free and reference-based judges, auto-calibrate LLM judges via APO, and A/B test in production at enterprise healthcare scale.

Chapters:
0:00 Introduction and what Abridge does
1:45 The most important workflow in healthcare: the patient conversation
3:22 Why trust is earned in drops but lost in buckets
4:27 Case study 1: clinical notes and why they are not just summaries
5:45 The real dangers: misattribution, hallucinations, upcoding, downcoding
6:55 How Abridge cut release cycles from 2 months to days
7:23 Migrating to LangGraph and LangSmith for evals
8:03 Building and auto-calibrating LLM judges with APO
9:42 Reference-free vs. reference-based judges: why you need both
10:55 The A/B testing approach most healthcare companies can't do
12:38 Case study 2: the Abridge Assistant agent
13:37 Design principles: air conditioning, agency, responsiveness
15:22 Eval criteria for a multi-step agent in clinical settings
16:39 Two takeaways: velocity plus quality, and why healthcare needs great builders

Resources:
→ LangGraph: https://www.langchain.com/langgraph
→ LangSmith: https://www.langchain.com/langsmith
→ LangChain Academy: https://academy.langchain.com