How Clay manages 300M agent runs a month with LangSmith

Clay's Head of AI Jeff Barg sat down with LangChain Co-Founder & CEO Harrison Chase to discuss how his team uses LangSmith as mission-critical infrastructure for observability, evals, and the agent development lifecycle.

Watch the full video to learn:
• What putting agents in production really looks like as you scale up to hundreds of thousands or millions of runs.
• How to think about agent quality at scale, and why Clay focuses on quality, throughput, and cost.
• How LangSmith helped Clay go from no visibility on inference spend to 99.5% cost reconciliation across providers.
• What's next for agents, and advice for teams scaling from zero to one.

0:00 How Clay thinks about AI: find, close, and grow
1:09 From chat completions wrapper to Claygent
2:02 The three agent categories powering Clay today
2:34 Running 300 million agent runs a month
3:20 How agent complexity changed Clay's dev process
4:06 How Clay measures quality: evals, deterministic checks, and LLM-as-a-judge
4:52 Staying model-agnostic with a metaprompter tool
6:01 How LangSmith fits into the agent development workflow
7:09 Why you can't catch everything before production
8:00 Tracing from day zero: the iteration process
8:35 Why Clay chose LangSmith over building in-house
9:27 Connecting a custom agent harness to LangSmith
9:44 The LangSmith features that matter most at scale
10:44 Who at Clay uses LangSmith (and how support uses it too)
11:12 Quantifying LangSmith's impact: cost reconciliation at 99.5%
12:18 How agents in production are changing — and what LangSmith needs next
13:15 Subagents, traces, and the future of self-healing workflows
15:06 Advice for teams scaling agents from zero to one
15:29 Agent memory: what's worked, what hasn't, and what's next
17:02 Closing thoughts

Extra resources:

- Learn about LangSmith: https://www.langchain.com/langsmith-platform
- Customer stories: https://www.langchain.com/customers
- Subscribe for more: https://www.youtube.com/@LangChain