AI Evals & Discovery - All Things Product with Teresa & Petra

Hi, folks. This is All Things Product with Petra Wille. And Teresa Kortz. And we're so happy you're here. Teresa, people had to wait one week, seven entire days for this to be published. Last week when we spoke, we talked about skills for building AI products, right? And you were already saying like evals is a big and important topic that we need to unpack for our audience. And so this is what we will be doing today. Teresa, what the heck are evals? What's the definition? Why should people care? Yeah. Okay. I'm going to give it. a 30-second summary of what we talked about last time in case people didn't listen to that episode. We started to get into what are the new skills required to build good AI products or features. We kind of broke it down into one, you have to know is AI appropriate for this problem. Two, we get into prompting in a way that's different from just regular chat GPT prompting, but prompting that's going to work across thousands of calls. We got into system orchestration workflows, agents. how to decomposing prompts into lots of little prompts. And then we got into evals. How do we know if the thing we built is any good? And then the last thing we talked about was ongoing maintenance and evals in particular require a lot of ongoing maintenance. Is it fair to say quality assurance for AI? I mean, in the like literal sense, is our product good? Yes. Okay. in the like pragmatic I'm following a test script and that's how I'm doing quality assurance? No. No. Yeah. Okay. So I'll tell you how I ended up on this topic. I built my interview coach. I started sending students feedback because I'm paranoid and I wanted to make sure my interview coach was good. I CC'd myself every time a student got feedback. And I started to notice some weird problems. Like sometimes the interview coach, it would identify a problem in the student's interview transcript. And for people that aren't familiar with my interview coach, it gives my students feedback on how well they conducted a story based customer interview. So they submit their transcript and then they get graded by my interview coach. Sometimes the interview coach would highlight like, you asked a really general question here, which is good. We don't want them asking general questions. And then it will like suggest that they could have asked something else instead. Sometimes in those questions, they're suggesting a leading question or a general question. So that coach is like, It kind of gets the concept, but then it makes a mistake. And so this, so then I was like, oh, well, clearly I have to fix my prompt. I'll just do a better job of explaining these things. And then I would release it and then it would fix one problem, but then another problem would pop up. And so it was like two steps forward, one step back, one step forward, two steps back. And I started getting like really concerned about like, how do I know if my coach is good? And I'd already been hearing this word evals pop up in the AI ML world. And I was like, what's an email? And that seems like an easy question, right? But it turns out it's not. So this idea is steeped in the ML world where what we did to evaluate ML products was we created data sets and they were they're often referred to as like a golden data set. It's a data set where we say, here's the input, here's the desired output and our that's like one piece of data. And then we're going to collect a lot of data. in theory represents the inputs we might see in production. So if we're trying to classify cats and dogs, our data set is going to have lots of types of cats, lots of types of dogs. And some mice, hopefully. And some mice and probably some horses and some zebras and whatever. Exactly. Edge cases, people, edge cases. And then we're going to define, usually in a spreadsheet, this is how most people started. Here's my input. Here's the desired output. That's a horse. Right? Or not a cat or a dog is probably more likely than the desired output. Not cat, not dog. And then we're going to, every time we make a change to our product, we're going to run all these inputs through our classifier. And we're going to compare the outputs to our desired output. And that gives us a score. So that's like the simplest eval we can think of. It's just I have a data set of use cases where I've defined the inputs. I've defined the ideal output. I can run my LLM app against all those inputs and I can score how well does it do. Okay, that's great. If you have your data set represents what you're actually going to see in production. So your data set needs to, your golden data set needs to look a lot like what you're going to see in production. Ah, and that is where discovery comes into play, Teresa, because now we need to discover how the world is really and what data we need to expect. Definitely. Definitely. Okay. So let's move into like an LLM example. I'm going to use a simpler one than my interview coach because I'll explain the problem I ran into with my interview coach. Let's take a simple one. Like let's say we have a meal planner app and it's just a chat bot. You tell it like, I need an idea for what to make for dinner. It asks you some questions. It gives you a recipe. Okay. What are your inputs? Like how are you creating your golden data set? Like what? Like a user could type in anything. Okay, I can go interview people and learn about like how they decide what to make for dinner and what questions they have and what constraints they have. And I'm going to start to get a sense for like, okay, well, some people have allergies. We need to have an input that represents like I need dinner, but don't... High protein diets. Yep. Diet preferences. Maybe they're limited to what's in their kitchen right now. So I can learn about a lot of use cases. What's in season. And I can start collecting these in my dataset. But it's a chat bot. Anybody can come along and enter anything. Yeah. Right? So my dataset is only as good as my known... What is the most poisonous meal to kill my stepmom? Yes. Yeah. So our dataset is only representing our known knowns. Yes. Right? Rumsfeld, Rumsfeld, hello. Yeah, yeah. Okay. Let's not talk about Rumsfeld. Okay. But it's in scale. It's in scale. Okay. I was creating an interview coach. My inputs were interview transcripts. Yeah, of course. What's my data set? Like, do I have to just wait and collect a lot of interview transcripts over time and create my data set? That's a problem. right because like does that mean i have to like let my interview coach run for 200 interviews in order for me to get a 200 data set and like for my first 200 students i don't know if it's good or not like this is the data somewhere maybe okay for some use cases so this is where synthetic data becomes an option a thing we can use llms to generate synthetic data but that's what i would have done immediately yeah but now i need to generate realistic interview transcripts. This is not easy, right? No. And the hard part is you need to wait. I think you need to give them weight as well. Right. So you would say like, so how I would do it, I would ask Chachapiti to create synthetic data, but then I still would use the filter of real life problems and interviews to categorize. Okay. And we, we saw roughly 10% of the interviews going down that route and 80% going down that route or something like that. Yeah, so I had some easy use cases. Like I only want my coach giving feedback on a story based interview. Yes. So in my data set. Okay, that's an exclusion rule. I need some story based interviews. I need some not story based interviews. Yeah. Then there's an issue of like in our class we do like little eight minute interviews. That's very different from a 30 minute interview, a 60 minute interview. So I'm like, okay, I need some interviews of varying length. because I have a lot of experience with interviewing, I know some interviewees are really good and gregarious and offer a lot and some interviewees are very terse and don't want to share anything. So maybe that's a variable. So I started, I can start to generate these dimensions. Dimensions. And I'll share, I took an AI evals class and in my AI evals class they taught us you can use those dimensions to feed to an LLM to generate synthetic data. Of course. I'm like, okay, great. So this is pretty good. But it's still not great. Like I have to trust that my data set reflects what's going to happen in production. So like this is great for getting to my V0, for my MVP. It is not great for like long-term high-quality production product. And this is where I started to dig in and like dive deep on evals. And I took a class and I started following the like ML engineers that are like the leading experts on evals. And here's what I uncovered. Do you want to know how they're getting better at evals? They infuse real data as soon as they have to have it? They look at their data. They're literally logging traces. So every user input and output. Which does make sense, yeah. They're storing it in a database. It's a future eval data point. And they're literally looking at it. Yeah. They're literally looking at it. They're saying this user entered this. This was the LLM response. and they're annotating it. Was that good or not? Yeah. So they're annotating real traces. They're having a human who has domain expertise, annotating every trace, like hundreds of them. I did this. I started with 100 interview transcripts. I ran my coach through it. I started logging what was working, what wasn't working. And then they teach. Okay, now look across your annotations and identify your most common failure. So I started to identify a failure mode. Sometimes the coach suggested a leading question. Sometimes the coach suggested a general question. Sometimes the coach suggested a question the participant already answered. Sometimes because I had split my prompt into seven prompts all grading different dimensions, sometimes an LLM would think the whole interview should be about their dimension. Right? So like... One of our dimensions is we teach people to set the scene. And then the rest of the interview, you're building the timeline. Well, the scene analyzer thinks the whole interview should be about setting the scene. Yeah, but that's when the orchestration comes back in, right? So one of my error modes was you expect the whole interview to be your dimension and that's wrong. Okay, so I looked at my data. I started to identify my error cases. Some of my errors, I could just make them go away by changing my prompt. Right? So I'll give a real simple example. Yeah, of course. Yeah, that makes sense. Because I orchestrate my LLM calls, all my LLM calls return structured JSON. Sometimes when you tell it only return structured JSON, it returns just structured JSON like you would expect. Sometimes it returns a tick mark first. Because in markdown, which is how ChatGPT responds in your browser, In markdown, a tick mark says the following block of text is code. But here's the problem. If you pass that string that is no longer valid JSON, you get an error. It's like a random semicolon somewhere. It was like one out of 20 times. Like 19 out of 20 times it would work perfectly. One out of 20 times it would fail. Well, I learned a trick. I use Anthropic for most of my interview coach. In your API call, you can tell the LLM your output, you can like pre-pend your output, what you want the desired output to be. So I can tell it your output has to start with a curly bracket. And then it returns valid JSON because it's not going to put a tick mark because it's being told you have to start with a curly bracket. Okay. Okay. So like I had to learn stupid tricks like that, but that came out of doing my error analysis and seeing that like sometimes it It starts with markdown. Okay, so those error categories I could make disappear by learning about them, fixing my prompt, learning some of these tricks. But there were always some error categories that just persisted. You'd make a change, some errors would get better, some errors would get worse. These are the categories where we're going to write code-based evals or LLM, judge evals. So here's how this works. Let's say I have an error category of that coach suggested a leading question. Okay. Now my user submits a transcript. My coach sends feedback. I log both inputs, the transcript and the response in my database. I get permission from the person to store their interview transcript. They both go in my database. Then my evals run. So one of my evals is looking for in that response, did the judge, did the coach suggest a leading question? Suggest a leading question. And so what I do for that eval, I take the LLM response. I make a list of all questions that show up in that response, which LLM does that. It pulls out all the questions. Easy. Another LLM looks at, is any of them a leading question? And if any of them are a leading question, that test fails. Yeah, of course the test fails. And which triggers then another part of the software rewriting the response? Hang tight, right? Right now we're just talking about evals. You're getting into guardrails, which we will talk about. Okay, so right now we're just talking about evals, right? So my judge evaluated this error existed in this trace, right? Another one of my evals is for did the judge suggest a general question? This is not an LLM as judge. So that first example is LLM as judge. My eval for suggested general question in that same list of questions, it looks for typical, typically, usual, usual, usually, general, generally. It's just code. It's just a code eval, right? And if it detects any of those words in the questions, it fails. Nice. That's a nice one. I love it. So across my interview coach, I have eight LLM calls. Each LLM call, I did error analysis and generated failure modes. For the failure modes that persisted, I wrote either a code-based eval or LLM is judge eval. Here's what this gets me. Now let's say I want to make an improvement to my coach. I can obviously have my fixed golden data set. that tells me my desired inputs and my outputs and get a score. But I also have all these other use case failure mode specifics evals that tell me, do these error modes show up in any of these traces? So now I'm a product manager. I want to run some experiments. What if I described a leading question this way and I changed my prompt? Should I release it to production? It depends. I got to run my evals and see, does this change perform better across all my favorite modes compared to what I have in production? Yeah, amazing. And now I can run all sorts of experiments and get black and white results. Is this better or not? I love it. And it's not black and white. Sometimes it'll be better on some failure modes and worse on other failure modes. But now I can use my human brain to make a judgment. Right? And then you still have it. Yeah. And then you already raised an interesting question. If I detect a general question, can I use another LLM to fix it before it goes to my student? So that's called a guardrail. When you run basically an eval in production before you respond to the user, that's a guardrail. And you can run some of your evals in production as guardrails. Generally, you don't want to do all of them because we're not running evals. Super expensive and performance heavy. It increases latency, it increases cost. Exactly. So what we typically do is we run evals on a percentage of our traces. Now, how do I know that my evals are good? How do I know that they're identifying my errors correctly? I need to have a human. grade traces and I need to compare my automated emails evals against my human graded evals and that allows me to score my evals and find my error rate on my evals. Okay, great. I've done that. So I can tell you how each of my evals performs. But here's what we know from some really interesting research actually by one of the instructors of the class that I took. There's this concept of criteria drift. So like we can't just let our evals run and call it done. Over time, as we see our use cases, our understanding of what good looks like evolves. Which means... That's what the LLM does, right? It evolves with all the data that it sees. So obviously you have to adapt the evals. So this means we could always uncover new failure modes. we might need new evals. Our judges might drift from human grading. It's a bit like owning a garden. So you're never done gardening. You're never done gardening. Right? So to release your feature, you built your first set of evals. But now that you're in production, you need to always be logging some percentage of traces, having humans grade them, having your evals grade them, and making sure your evals are still performing the way that you expect. This is a lot of overhead for AI. Yeah. And this is what really from just listening to you, Teresa, what I thought the entire time is whenever people think, say like, we now can spin up these products in hours. I'm always like bollocks. You can't. I totally buy. You can test ideas much quicker than before because it's then it's prototypes. You don't put them in production. You don't need to consider all these things. Evaluations maybe not so much important, right? So in discovery where a product manager is using AI to prototype stuff and to get an idea in front of an executive or a customer ideally or something like that. I totally get it. But. products because I'm still working on my product leadership wheel assessment tool that we already talked about as well I'm still not planning to launch it because I don't want company to use it as a big scale assessment anyways it's that's an amazing use case for me to play with AI and currently doing a lot with lovable and stuff like that but releasing that into production so many more things to consider and it takes really as you were saying now you need human really like humans looking at your data yeah so people not longer do content moderation maybe on behalf of mark sockerberg but now they need to look at prompts and how responses uh went and grading them and all these kind of things so it's yeah it's not as easy as people make it look Yeah, okay. So I'm going to wrap this up a little bit here. Yeah, please. So I got introduced to evals and this like more in-depth way of doing evals, like through error analysis and writing code evals and writing LLM as judge evals, not just data set evals. And it blew my mind. I was like, okay, I can see how this can give me confidence that my AI product is good, right? I can see how I'm building in. infrastructure to evaluate my product. I'm like, wow, this is a lot like discovery. I now have a really good feedback loop. But I also know from discovery, all of these steps in this process are only as good as my understanding of the customer. Can my human graders say this is good if they don't know what a customer expects? Can I generate the right synthetic data if I don't know what my customers want to put in? And so what I'm really starting to think through is what does discovery on AI products look like? How does discovery support how you write your prompt? How does discovery support how you orchestrate your system? How does discovery support how you do your error analysis and what you pay attention to in a trace? And how does discovery support what evals you bother writing and how you judge something is good or not? And I'm really curious about if I really push the envelope, how do we get the customer to give us that feedback? Right? Yeah, but that to some extent is possible, but still you have to orchestrate so many things in the background. Yeah. If, by the way, people, you want Teresa to write that book, then just send an email to Teresa at producttalk.org. Did you really just give her email address to all of our listeners? I think it is well known across the internet, Teresa. That's not a secret. Because the thing is, really, and I know you were launching a blog post at first, and I really think that's a great idea. And we have this podcast episode recorded, but there will be so many things that product people will need to wrap their head around. Okay, this is probably... In this discovery slash AI products realm. This is probably a good place for me to preview what's coming next. So I've been using July and August. I know we're now into September. It might even be live by now, but with the time that we're recording, it's not. I've been playing with how do we help product teams get really good at AI products? What does that look like and what role do I want to play in that? And of course, because I teach courses, the first thing I thought of was like, should I do a course? And then I was like, no, I'm not an expert in this yet. I've built one AI product. I'm talking to a lot of people who are building AI products and learning from people. I have my own personal story and I did a little webinar to test this out. I did an hour where I talked about how I built my interview. I saw that. Yeah, amazing. That will definitely be a blog post by the time this goes live. So we'll put that in the show notes. And I realized like there's two ways I can help. And this is my current plan. We'll see if by the time this goes live, if it's still the plan. I am going to write a little mini blog post every Monday. I'm going to treat it like my weekly demo. I'm just going to share what did I personally learn building AI products this week. So for anybody who wants to follow my personal journey as I nerd out on this stuff, that's going to be the Monday post. And then this second piece I really want to do, but I'm still testing assumptions around it. So we'll see if it comes together. I want to start another podcast. where I interview cross-functional product teams about how they built a real AI production product and feature. And we get into all these steps and what they did and who did what and how we're collaborating on this. Because I think we just need a lot of examples. Yeah, we need real life stories. That is really what people will need in the next few years. So I've already talked to four people that would be great guests. What we're working through is company permission. What can they share? Is it still going to be a good story? So hopefully that comes to fruition. I'm going to run both of these through my sub stack. So I have a sub stack product talk daily. You can find it at producttalkdaily.substack.com. Monday posts will go out there. Thursday podcast episodes will go out there. Product talk Wednesday posts will be deeper dives. Like the model I explained in last week's episode. We're doing a big blog post about how people are using Lovable not to build personal products like Lenny just released recently, but in our product work. So I'm basically diving deep on AI. You're going to see a lot on AI from me across lots of channels. And from this podcast episode, people can tell... you're a geek. So it will be really a deep dive. Yeah, I love it. I love it. Turns out this is what I've been doing for the last four months. Yeah, that's good. That's good. Because why not? And somebody has to do it. It's really fun. Thank you, Teresa. Is it already thank you, Teresa time? I think so. I think we can wrap up. Okay. Dennis, thank you, Teresa. See you next week.

Building AI products isn’t just about clever prompts and orchestration—it’s about knowing if what you’ve built actually works. In this episode, Teresa Torres and Petra Wille dive deep into AI evals: how they’re defined, why they’re essential, and how teams can implement them to ensure product quality.

Teresa shares her journey building her Interview Coach tool and the hard lessons she learned about evals along the way. From golden datasets and synthetic data to error analysis, code-based checks, and LLM-as-judge methods, you’ll walk away with a clearer picture of how to measure and improve AI products over time.

What you’ll learn in this episode:
✅ What “evals” actually mean in the AI/ML world
✅ Why evals are more than just quality assurance
✅ The difference between golden datasets, synthetic data, and real-world traces
✅ How to identify error modes and turn them into evals
✅ When to use code-based evals vs. LLM-as-judge evals
✅ How discovery practices inform every step of AI product evaluation
✅ Why evals require continuous maintenance (and what “criteria drift” means for your product)
✅ The relationship between evals, guardrails, and ongoing human oversight

Resources & Links:
Follow Teresa Torres: https://ProductTalk.org 
Follow Petra Wille: https://Petra-Wille.com

Mentioned in this episode: 
How I Designed & Implemented Evals for Product Talk’s Interview Coach by Teresa Torres: https://www.producttalk.org/2025/09/interview-coach-evals/
Teresa’s Interview Coach: https://learn.producttalk.org/course/story-based-customer-interviews
ML (Machine learning): https://en.wikipedia.org/wiki/Machine_learning
Story-Based Customer Interviews - On Demand course by Teresa: https://learn.producttalk.org/course/story-based-customer-interviews
LLM (Large language model): https://en.wikipedia.org/wiki/Large_language_model
AI Evals for Engineers and PMs course (get 35% off through Teresa’s link) on Maven: https://maven.com/parlance-labs/evals?promoCode=torres-35
V0: https://vercel.com/docs/v0
JSON (JavaScript Object Notation): https://en.wikipedia.org/wiki/JSON
Anthropic: https://www.anthropic.com/
The Product Leadership Wheel - A Framework for Defining and Growing Product Leadership at Scale by Petra Wille: https://www.petra-wille.com/plwheel
Lovable: https://lovable.dev/
Behind the Scenes: Building the Product Talk Interview Coach by Teresa: https://www.producttalk.org/2025/08/customer-interview-coach/
Previous episode: Building AI Products: https://youtu.be/QYf3Y1Zb83Y

Coming soon from Teresa:
Weekly Monday posts sharing lessons learned while building AI products
A new podcast interviewing cross-functional teams about real-world AI product development stories

Have thoughts on this episode? Leave a comment below.
#AllThingsProduct