Observing And Testing CX Agents | Interrupt 26

At Interrupt, the agent conference by LangChain, Carlos Pereira from Cisco's Customer Experience team showed how you close the loop between production feedback and code at 16 million customer interactions per year.

The core problem with scaling agents: once adoption climbs, feedback volume outpaces your team. Every thumbs down, every confused user, every low-confidence routing decision is a signal. If you treat it as noise, or if your team becomes the bottleneck processing it, adoption drops. This talk walks through the system Cisco built to take a user's thumbs down all the way to a merged PR, with AI handling triage and diagnostics and humans in the loop only where decisions matter.

Observing And Testing CX Agents | Interrupt 26
0:00 Introduction and Day 2 context
0:49 What today covers: observability, testing, and support
1:11 Cisco support at scale: 16M interactions per year
1:34 When network outages hit: why this matters
1:54 CX methodology and team structure
2:16 Evolution: from chatbot to autonomous teammate
2:51 Today's question: closing the production feedback loop
3:23 The approach: continuous feedback loop, not a ticket queue
4:11 Every signal matters: thumbs down, errors, and confusion
4:41 Why humans become the bottleneck at scale
5:57 Signal capture via LangSmith traces
6:12 Triage agent: LangSmith MCP and Jira MCP in practice
6:55 Code agent: clustering and diagnosing issues
7:35 Human in the loop: only on writes, not reads
7:56 Proactive and reactive feedback pipeline
9:20 Treat evals like tests, not experiments
10:56 MCP as the integration layer
11:21 Human oversight only on writes
12:47 Lessons learned
13:53 Observability is your new bottleneck
14:05 Close the feedback loop with agents
14:36 Evals are infrastructure, not a side project
15:34 Support use case: Cisco technical support
15:48 Live example: enterprise network assessment
17:04 2,176 security findings: where do you start?
18:13 Semantic routing for ambiguous prompts
19:33 Parallel pipeline with guardrails
20:55 Tracing every step with LangSmith
21:36 Self-learning routing system
21:54 Observing LangSmith itself with Splunk
22:22 Closing

Extra resources:
• Everything we shipped at Interrupt: https://www.langchain.com/blog/interrupt-2026-overview
• Meet LangSmith Engine: https://www.langchain.com/blog/introducing-langsmith-engine
• About LangChain: https://www.langchain.com/