From Noob to Automated Evals In A Week (as a PM) w/Teresa Torres

Welcome everybody. We're really excited about this talk with Teresa Torres, who was a student in our first cohort. Teresa Torres is a legendary person in the product management space. She has trained, I think like over 70,000 people or something like that on how to do product management effectively. She's the author of this book, Continuous Discovery Habits, which is right behind her. You'll see it. You should definitely check that book out it's very it's like one of the canons in product management from my understanding i'm not probably managing myself but every product manager tells me that the reason this talk is so exciting the reason why you should pay attention to this is hey like teresa she's not coding every day or anything like that and wasn't like necessarily doing evals but she had her own approach combined with her product management expertise And she has a unique take on how she went about things and has found a workflow that she likes. And she's going to share that today. So I'm going to hand it off to Teresa to get her started. All right. I'm really excited to do this. I really enjoyed this class quite a bit. And you're going to see I learned a ton and was able to put a lot of it into practice right away. And I cannot emphasize this enough. I forget when the first cohort started. I think it was in June, maybe May. I barely knew what an eval was. Like, and we'll talk about that in my talk. I am going to share some slides, but I'm happy to keep this very conversational. So please feel free to jump in, Hamill or Shreya, with questions. And we'll just dig in. I'm going to start by giving a little bit of context about the product that I'm building. Campbell mentioned some of this in my intro, so I'll touch on this briefly. Like my day job, I work as a product discovery coach. It means I help product teams make better decisions about what to build. You might be, some of you I know are familiar with my book. We have a wide variety of courses. In fact, I saw Letty is here. Letty is currently in our continuous interviewing course and has gotten to experience the continuous, the interview coach firsthand. But basically because I primarily am an educator, I When I started to think about generative AI and how it might impact what I do, I was thinking about it from a teaching standpoint. And that's the part that's relevant here. So I found myself in March, I actually had to have ankle surgery and I was like on the couch for months and I was like, how in the world am I going to use my time? And I started this with this question of what does generative AI make just now possible? And I was asking this in the context of our courses. And there was one course in particular I thought this would help a lot with, which is our continuous interviewing course. So in this course, students learn how this is customer interviews, so not job interviews. And in this course, students learn how to ask the right questions. They practice active listening techniques. We teach a very specific type of customer interview where they're asking the customer to share a specific story about their past behavior. And there's reasons for this. This is a way to collect reliable feedback. I'm a little bit of like a research methods wonk. So I'm like trying to look at what's the easiest way to teach product teams who are not researchers effective ways to get customer feedback. And in all of our classes, they're very practice oriented. So like in this class, students come to class and they interview each other and we give them a rubric to help give each other feedback. Right. So I'm a big fan of deliberate practice. We practice a skill, we get feedback. We then know exactly what to practice. There's one challenge, though, like when students give each other feedback, they're not really experts, so they don't give expert feedback. And so the problem I was looking to solve was like, can we help students give each other better feedback on their practice interviews? And that led to this question of can we model what good feedback looks like? And so this is the tool that I built. I built an interview coach. where students submit their interview transcripts and they get very detailed feedback on what their how they did on conducting an interview and the coach actually this is actually what you get from the coach um and the coach this is actually a partial response but we the coach is basically evaluating an interview transcript on four different dimensions and this is from the rubric that we teach in the class so that we're evaluating did they ask a story-based question did they ask the participant to set the scene that's the tune that you're seeing here there's also a dimension about building the timeline how well did they get the participant to tell their story and then did they do a good job of redirecting generalizations and you can i'm going to dive into this a little bit because a lot of my evals are about this coach feedback so this will give you some context so you can see here for each dimension they're getting a score the score can be keep practicing getting it or great this coach is pulling out real excerpts from the interview transcript and then giving them a tip. So this might be, this was really good. It might be, here's some room for improvement. And it's doing this across all four dimensions. So there's two sections of this email we're not seeing, but it's basically excerpt, tip, excerpt, tip. Okay, so that's the interview coach. So let's get into, as I start- When you see dimensions, is it that those, these are like different stages, opening with a story-based question and setting the scene, are those the dimensions? Yeah. So what we're looking for in the transcript is, did they ask the participant to share a story? We teach that they should ask them to set the scene because this helps with memory recall. So it's a cognitive tip to help get a more reliable story. And then the building the timeline dimension is, did they help the participant go step-by-step through their story so they're learning what actually happened? And then the last dimension is, if I ask you, tell me about the last time you watched Netflix, You might start to generalize about what you usually do on Netflix. And so my job as the interviewer is to redirect you back to the specifics of your story. And so that's the fourth dimension that we're evaluating on. Interesting. Yeah. I need to read the book too. This is already very interesting to me. Awesome. Okay. So as I was building the interview coach, I literally was like vibe checking the coach's responses. Some of them looked good. Some of them had some problems. And I started to get concerned. Like, how do I know if my interview coach is any good? I would make like... So what did you have in the interview coach? Was it like transcript of your book? What information like was your initial version? Yes. So because we have a course on this topic, I already had a rubric for how to evaluate an interview. And so I started with the rubric that we use in the class. And then I included a lot of the instructional content. that we provide to students in the class to the LLM. And like my very first prototype was in a CLOD project. I just uploaded a bunch of files to CLOD and I was just like, here's an interview transcript, give me feedback. It obviously has iterated and it evolved quite a bit from there. But basically the coach, the AI tool, the coach gets a full interview transcript and then it has a lot of context about what's a good interview, what these four dimensions are. it started as one prompt it's now like depending on the content and the context and what errors uncover it's anywhere from five to nine llm calls um so it's a pretty complex workflow and i can imagine one day it will be an agent like what i would love to do is to pull in a student's learning history and then adjust the feedback based on where that student is on their learning journey and get and move the coach to more of an agent model where it's choosing what dimensions to evaluate on based on the student's history but i'm not there yet okay so like in my earliest days building this so thank all by the way all this is fresh i started this on march 23rd and i really had never built an ai product i was using all no code tools at this point and i just was making i had one prompt like it started as one prompt and i was i didn't even know what temperature was when i'd started and so as i was making tweaks I would see some things get better and some things get worse. And I was like, this seems insane. Like product discovery is all about feedback loops. That's literally what I teach. And so I was like, oh my God, what is my feedback loop? Like I need a more systematic way to evaluate this. And you're all in an evals class, so you already know the answer is evals. But I got stuck and you might know, you might understand why. Like I started to open AI. they talk about evals and it sounds like it's just a bunch of input data and i was like yeah but i don't have a bunch of transcripts and maybe i could synthetically generate transcripts but let me tell you how hard that is like it sucks and so i kept looking and i found like this python package and it was interesting to me here is it started to talk about evals as code and i was like oh interesting that's like it interesting and then i came across a mon con's blog post on lenny's newsletter and i was like oh there's different types of evals and now i got sucked in and that's we're going to skip this you guys know what evals are that's when i found this class and i will tell you in the very first session when this model was introduced like light bulbs went off in my head because this is the scientific method i teach the scientific method and i was like okay i'm in let's do it And so over the course of the four weeks of class, I literally, instead of doing the homework, I just applied each lesson to the interview coach. And this is how I work. I'm a very goal-directed learner. So I was like, I have a project. I'm going to take everything I'm learning and just apply it to my project. And so again, some of these slides were for a non-experienced audience. So I'm going to skip past them. So I started by collecting my traces in Airtable. So why did I use Airtable? This is a tool that I'm familiar with. i was already had plenty of new to learn and this is going to be a theme of this talk like i try to limit how much i have to learn in each step so that i don't get overwhelmed and so i was like okay i see that there's these eval tools but they look really overwhelming to me i'm just going to store everything in air table and so that's what i did it's an excellent approach i highly recommend that people do this is hey don't try to swallow these tools everyone wants to go for tools but yeah stay where you're comfortable with in the beginning for sure so i think I had, God, where did my original data came from? I think I had about 15 interviews that I conducted. I think I had about 10 other interviews that some of my instructors had conducted. And then I think I had about 10 interviews that some of our alumni let me play with. I was developing this tool. So with small data sets and I actually used air table interface as my very first annotation tool. So if you're not familiar with air table, like it's a database basically. And then. It gives you these like UIs to construct viewers. And you can see here, like I have a trace down here at the bottom. I have an annotation box. I have my failure modes. And then I can mark it as reviewed so that it doesn't show up again in my annotation tool. Dead simple, right? And if I go back, you can see like I've got a bunch of annotated traces. I've got failure modes. These screenshots are from my very first round of doing any of this. This is literally during the class. And I'll tell you, when I saw this, look at all these failure modes. I was like, holy crap, my coach sucks. And what's funny is that I thought it was pretty good. Like when I was just vibe checking responses, I was like, oh, it's pretty good. Yeah. I was embarrassed at this point. This is amazing. I love this so much. Okay. So I then sat down. I remember this vividly because it was like a Sunday evening. I'm sitting on my deck right next to my husband, who's also on his laptop. We're both nerds working away. And I was trying to design my first set of evals so i was literally like whenever i code i didn't talk about my coding backgrounds as an undergrad i took some computer science classes i did a cognitive science program where computer science was a big part of it i my first couple jobs i was also a front-end developer so i did javascript development before there were any frameworks i did not code for 15 years And then maybe longer. And then about three years ago, I started coding again to automate a lot of my business logic. So I run cohort-based courses. Running cohort-based courses requires a lot of data management. And so I started coding to work with that stuff. I'm not really an engineer. I had never worked in an IDE. I didn't have a dev environment and a prod environment. I didn't know how to do these real things. And so I actually chose to work in AWS's serverless environment. because like i didn't want to manage a ec2 instance like i just wanted to like execute code and move data from this app to this app and so i started writing node.js lambda functions and then orchestrated them with step functions so that was my experience when i started this product and you'll see this influences a lot of my tech decisions okay so it's sunday evening i'm like okay this is in the middle of the first cohort i've identified my top failure modes and i'm like i need to write my first evals and whenever i write code i always start in pseudocode and i just start like in a text document and i literally was in an apple note and i was like writing out json and i don't know if you guys have ever used apple note but it turns quotes into smart quotes which breaks your code and i was like banging my head against the wall and my husband goes why don't you use a jupiter notebook and i was like what what's a jupiter notebook so i literally asked chat GPT. I'm a total beginner. What's a Jupyter notebook? For those of you that aren't familiar with notebooks, this is a Jupyter. This is a screenshot of a Jupyter notebook. This top part up here is Markdown, and then my code is here. Here's what this allowed me to do. I started in a Markdown cell and just brainstormed like different ways to do an eval in kind of pseudocode, and then I could try to write that in my cell block and actually execute it. and then see the results. And then what I realized, which was really great about this, is like for one eval, I might have three different ideas about how to implement it best. I could do all three in a different cell block, run them, and then compare their output. And so that was awesome. I was like, okay, I'm sold on notebooks. I'm going to show you two of my first evals that I ever created. So this one, so we need a little bit of context. Remember, the coach is taking in a transcript. and it's sending you detailed feedback on that transcript. And sometimes it'll suggest to you, it'll be like, Hamill, you asked a great question here, but you could have asked this follow-on question, right? So sometimes the coach will suggest a question to you. The problem is sometimes the coach's suggestions are wrong. So there's two failure modes that came up where sometimes the coach would tell you, you should ask a leading question. Like the question is suggested to you. was a leading question. And if you're not familiar with this concept, a leading question is just a question that implies the answer. So it's like you're leading the witness. That was one of my failure modes. And then my other failure mode that was really concerning to me is sometimes the coach would suggest a general question. So it would say something like, you should have asked, tell me about your typical morning routine. Well, we don't teach that because that leads to really unreliable responses. We teach, tell me about your morning routine this morning. So the specific, not the general. And so for someone that's new to notebooks and stuff, like it seems like you organize your notebooks quite well. You have like headings, you have a little bit of documentation and you kind of print it out to show yourself what it does. This is like, yeah, really good practices. So here's what I'll tell you. I'm not an engineer day to day, but like I did. I took a lot of computer science classes at Stanford where they hammer like elegant code and well-documented code. And like, I feel like I have a pretty good foundation in like good engineering practices. And then as a product manager, I always worked at teeny tiny startups where like I was involved in all the engineering decisions. So I think I'm a very technical product manager. And like actually now, like four months in, I'm ready to call myself an engineer. And you'll see why as we get further along. It's actually been really fun, like embracing my engineering skill set. Okay, so these two evals, these two error cases, the coach suggesting a leading question and the coach suggesting a general question, as a teacher, were catastrophic for me. These are two of the things that students really struggle with, and I did not want the coach reinforcing it. But if you look at this code, this one on the left looks really complicated. It is not complicated at all. What's happening here is this is a prompt where I'm explaining to an LLM what a leading question is. And then it gets sent a list of questions. And it's being told if any of these questions are leading, fail the test. And if none of the questions are leading, pass the test. All this code down here is just Python to call a production Lambda function. So you can ignore all that. So this is an LLM as judge. And I iterated on this a lot. I tried like sending all the questions at once. I tried sending one question at a time. I tried sending the full coach response versus just the questions. So that's what I mean by I tried different strategies for how do I get this email to be fully aligned with my human labels. And then this email, the general question actually banged my head about how do I teach a LLM what a general question is? Do I have to like. give it the whole context of a specific story and the human tendency to generalize. And actually I realized that there's just a small set of keywords that tend to indicate a general question. And this turned into a code assertion. I'm just looking for red flag words in the questions the coach suggested. I've extended this list of keywords over time, but what's funny about this very elementary eval is it has a really high alignment rate. Like it never has false positives and it catches almost everything I catch. I'm so happy about this. You nailed it because what you want to do with evals is you want to have the right cost benefit. And the fact that you had this intuition like, hey, this can be a code base eval. A lot of people would miss this, honestly. They would say, oh, I don't know. Let's have LLM do it. You had the good insight of making a code base eval. Okay, I'm glad you highlighted this because this was the moment where I was like, holy crap, who's going to do this in real life? Like product managers, designers, engineers, like what's the right rule? Because the reason why I could identify this insight is because I've been teaching this stuff for years and I see the mistakes that students make. Like this required domain expertise. And so if I just left this to an engineer who wasn't me, I don't think this would have happened. And so this is still an open question for me as a discovery coach that teaches cross-functional collaboration. to product teams, I'm like, oh, we got to get good at working really closely together because not every product manager is going to write code like I did, but I think they need to be involved in the design of their evals. Definitely. I can almost guarantee you with almost 100% certainty that a engineer would not have, would have tried to make an outline judge out of that. Yeah, totally. Yeah. So these two are literally the first two evals I wrote. I love them because they're still in production today. They're still really valuable. I've actually been able to reduce these errors quite a bit. And so this sold me. And I did this on that first Sunday night. Now, I wrote these on that first Sunday night. I didn't have any infrastructure for how do I run them against all my traces and how do I score my judges? So we'll get to that. Okay, so the other story I didn't tell you is on Sunday night when I started all of this, I had never even seen Python. But I think my husband works with Python and he might have told me that Python has all these visualization tools. I think he did. And also in class and in the Discord, everybody was talking about all the cool Python visualization tools. So my notebook is actually a Python notebook, which still blows my mind. And how did I do this not knowing Python? I literally had ChatGPT open. and was like here's what i'm trying to do can you help me figure out how to do this it would present some code i won't wrote i won't run code that i don't understand so i literally would go line by line and be like what does this mean what is this doing and then once i understood it i would copy it into my cell block and run it okay so chat gpt plus my python notebook helped me import like my traces so i exported a csv file from air table i imported that csv pile file and used like pandas to look at it and i created a cell block in my notebook that ran all of my evals against at this point i had one two three four five evals against my traces and oh the other thing i did in my notebook this i think might be helpful for people these two evals they don't actually take in the transcript I actually do some transcript processing to help make my evals more efficient. So one thing I noticed is a lot of my failure modes were related to the questions the coach was suggesting. So I have a helper function that it's just code that like takes the JSON response from the coach and pulls out all the coaching tips. That's just code. And then I have another helper function that I think is an LLM that pulls out all the questions from the tips. So then my evals don't have, especially my LLM is judge evals, they don't have huge input tokens. I've just constrained the context to what that eval needs to assess what it's assessing. And actually this has saved me a ton of money and makes my evals really fast because when I'm testing and iterating, I can run all my helper functions once for all my transcripts. And then as I change an eval, the input data isn't changing. So I don't have to rerun all those helpers. and those are like the expensive llm calls whereas my evals end up being a lot more lightweight and a lot cheaper this is amazing it's so amazing because not only are you saving money but like you're also making the eval better if you're constraining the context context to like only what it needs to know if you've done it's pretty advanced and that's really great that you like did that work and people would call that quote context engineering the new one you have been doing context engineering from the very beginning which is excellent yeah what's funny is that this tapped into i did this originally just as like decomposition i was like oh this eval and this eval need the same stuff i'm going to separate that into its own function and it was just like a separate cell block in my notebook and then i realized that oh this enables different experiments i can run Like, what if I give this eval the full coaching output? What if I give it just the questions? What if I give it one question at a time? And that's what really allowed me to tweak and figure out which strategy was most aligned with my human labels. And I literally did all of this in a single notebook. Okay, so then I look in my notebook. I worked with ChatGPT and I was like, okay, I have evals. I have the CSV file. How do I run all my evals against all my traces? It helped me create a code block that does that. And that code block outputted what you're seeing here. And this is basically my eval results. And this was my very first run. So I have one, two, three, four, five evals, and the traces are the rows. And you can see at a glance, I can start to see how I'm doing. So those first two error cases that I was most concerned about, they're happening pretty frequently. Like I think about 20 to 25% of the time. And then you can see in this middle one, it looks like, wow, this fails all the time. And this one on the right seems to be a huge problem. But as I dug into this, I started to realize some really interesting things. First of all, this one on the right, when I created this eval, what this eval is, is I have four dimensions, right? Sometimes a dimension is greedy. It expects every question in the interview to be about its dimension. So my setting the scene feedback would be like, This is a terrible setting the scene question, but it was a really good building the timeline question and the interviewer didn't do anything wrong. And so this error mode is it expects too much setting the scene. So like they asked good setting the scene questions and then they moved on to building the timeline and the coach was critical that they asked building the timeline questions. And so it looks like this is happening a lot. But what I did to dig into this is in my notebook, I have tools that allow me to look into each of these failures and to give and to examine the reason the LLM gave for why this failed. This one actually is a code eval, so that's not quite right. But I can look into the output of the code and then I can look at the individual trace and my human labels. And again, I'm doing this all in my notebook. I don't have screenshots of that, but I'm doing this all in my notebook. So I like... Basically, as I dug into my data, as I had questions, I would go to ChatGPT and be like, okay, it looks like my evals gave this and I gave this. Can you create a cell block where I can compare my human label to the eval label? And so then I just iterated with ChatGPT to turn my notebook into a data analysis tool. And what I found here in this last column, it actually was a problem with my human labels. like my eval was better than my human judgment and it's because to write a specific eval for this error mode i had to get a lot clearer about what i was evaluating and i like codified that in code but i hadn't codified it in my human label like rubric and so this was a case where the code was better than i was and so like i had to change the way i did human labels which i thought was really interesting it's just so amazing on so many levels the One thing that really impresses me is you actually took on the mindset of a data scientist. Pretty good, like putting on the hat of, okay, let me investigate the data. Let me look at it in different ways. Let me figure out what's wrong. Like really dig around with this sort of investigative approach. Yeah. And this is, I think where notebooks shine. i wrote little pieces of code to like visualize things a little differently and look at different data and in some ways it's throwaway code but it's not because it's reusable for every eval i work on so i just see at the end i'll share the notebook i use now i basically created like an eval investigation notebook that i use anytime i'm working on an eval so the other thing that came up here is like this middle one is actually the reason why it failed so much is because i had a bug in one of my helper functions and so That was also interesting to see like how this data really exposes where like something you're doing may not be working the way you expect. And then I did score these evals. So these are my also in my notebook. I basically asked ChatGPT to write me a Python function that would compare my human labels to my evals. This was my very first run. This was before I did any sort of iterations to align it. It's pretty hit or miss at this point. And I don't even, I can't even, this was so long ago enough, I can't even tell you what this blank line was. I think it's that I had human labels that I don't have emails for yet, but I'm not totally sure. So up until this point, this was that Sunday through Thursday. I did all of this in four days while taking the class. And I was really surprised by how far I could get. And a lot of this was because I just leaned on ChatGPT as a thought partner really heavily. Okay, so now a week in, I'm like, okay, I guess I'm a notebook Python developer. Let's keep going. And here's what this did for me. I think I already talked through some of this. Sorry. Let's see if there's anything new. Really what this did for me is these two error cases were my most concerning, and it really helped me size the problem. Like, how big of a problem is this? And that was really reassuring, because I know when I can measure something, I can improve it. It helped me find bugs. I already talked about that. And it helped me get further clarity on my rubric. That was this setting the scene one. So I go back to this question. I was like, okay, now I have a really structured way to ask, are my changes having an impact? And literally for every model change, prompt change, temperature change, chunking strategy change, or now I've gotten into workflow changes. I have a fast feedback loop. I basically make the change. I run my emails on my dev set that's based on production. And I run. my evals on my dev set based on my change. And I literally look at this table and it's never like totally better or totally worse. It's always these evals got a little better. This one got a little worse. And I have to make a judgment call about is this better for and should I release it or not? And I cannot emphasize this enough. Like as someone who teaches fast feedback loops, this just felt so magical. I was like, oh, I can actually measure a non-deterministic product. This is awesome. Okay, so what does it look like today? Today I work out of VS Code. The reason why I switched to VS Code is I am a heavy Cloud Code user. Like, I treasure my Opus minutes with Cloud Code because it designs everything for me. I want to clarify this because for the non-engineers in the room, I am not Vibe coding with Cloud Code. Like, I'm not letting Cloud Code... just produce a whole bunch of code that I don't look at. That terrifies me. I still have this philosophy, if I'm going to execute the code, I better understand it. So I pair program with CloudCode. We talk about what's the best way to do this and how should I implement this? And I don't like that pattern. What are the other patterns? And so that's how I get a lot of work done. And sure, CloudCode might write most of my code, but it's really important to me that I understand all of it. And we'll talk about why in just a second. But basically the way that I work is I, you can see here, I don't know if it's very visible, but like I have the whole evals directory in my project. I run experiments. So I have different directories for different experiments that I'm running. I had Claude Code write my AB test runner where it's basically running all my traces against what's in production versus my change. I keep track of, I have evals for the cross dimensions. I have evals for individual dimensions. So this is. growing into a gnarly beast but i love it what you're seeing here on the right is my notebook this is my current notebook that i work out of and actually this screenshot this in this notebook is this little tool where i can actually select a transcript and it shows me what my helper functions extracted and then what each of the evals returned and so this is like my magical diagnostic like what went wrong with this trace tool which the fact that i can just do this all on a notebook still blows my mind the level of joy that i have in this talk keeps increasing exponentially as you keep talking it's quite fascinating and so the thing that is really great here is like not only do you use notebooks for data analysis you create your own widgets and little user interfaces in the notebooks which is quite an advanced thing i want people to know you can do that okay so i didn't know i could do that clon code did that for me like i'm good at i'm really good at describing what i want right it's okay i see this error in this trace but you didn't tell me the trace number like how am i supposed to go investigate they'll be like oh let me add the trace number i'm like okay i have the trace number but i don't want to go to air table to look it up and i have a cache with helper functions and you let me look at the cache but just put it all in one place so i just demand things from cloud code and we work on it together and i actually If there's anything you take away from this talk, it's that you don't have to know how to do any of this because LLMs are really good guides. But I have Vibe Coded too, and what I'll tell you is just make sure you understand what it does. Because every time I give Cloud Code a little bit of a longer leash, things go horribly wrong. So like, I am pair programming with Cloud Code. Cloud Code is not just generating a bunch of code with me. For me. That's great, because I think a lot of people, they try to... eat too much like they try to power the element like eval everything without looking at the data you had a sense of okay let me kind of work with ai to understand the problem deeply not let it just look at my traces and do my evals for me which is like what a lot of people try to do and it doesn't work yeah i think something about this class like resonated with me so much at like my core that I just wanted to learn it really well. And I think also it's why I didn't want to use the eval tool. Like I figured if I rolled my own, I would know it inside and out. And even now, like even on the other side of this, I've been watching the eval videos from this second cohort and I'm still like, it just looks more complicated than what I have and what I have is working. So I'm going to stick with it. The other thing that's different today is my notebook has graduated. So before my first notebook, All my eval code and my data analysis tools were in one notebook. Now my evals are part of my normal repo because I run some of them as guardrails in production. I run them on a percentage of traces. And so now my notebook is just a data analysis tool and it can execute my evals, but it's executing my production code. So they're no longer commingled. And so in my notebook, I have some tools that allow me. The first thing it does is it allows me to run all my helper functions and cache them. So those are my expensive calls. I can do that for my dev set once. And then because my dev set isn't changing very often, the only time I ever have to rerun those is if I change a helper function. And that's been huge. And then I can rerun specific evals. I can rerun all my evals right from my notebook. I do my analysis and visualization right from my notebook. So this is a run I did yesterday. This is how I'm currently doing on that general, suggested general question. You can see I've gotten this error. It's almost non-existent. And actually when it shows up, it's showing up because there's some ambiguity. Like the participant starts by asking a specific question and then falls back to a general question. But literally having access to this data and his visualizations. has allowed me to iteratively like dramatically reduce the errors in my interview coach. And then the notebook also has like diagnostic tools. Okay, what's going on with this trace that failed? Can I go look at why it failed? Is it a problem with one of the helper functions? Is it a problem with the eval itself? Is it a problem actually with the trace and it's a real failure? And I can do all that right in the notebook. The other thing I can do in the notebook, which I don't have screenshots of, is probably because one piece of it is currently broken that I need to fix, but it can pull in my annotations and then compare my annotations to my evals and then score any of my LLMs as judges. I just changed my annotation output, and so that part of the notebook is currently broken. I need to fix it. But it's really nice having all of this in one place. And again, I just want to say... These notebooks are so well organized. They have sections, they have documentation, and that's not just aesthetics. This actually helps when you're using AI. Okay, so I will say Cloud Code created this notebook for me. And because Cloud Code created it, and I had to work my way through understanding it, I would regularly update the markdown because I just didn't trust that since I didn't write it, that I would remember it. So like a lot of the documentation is for me because like I come back to the notebook and I'm like, which cells do I have to run before I can run an eval or what's going on with the cache? And yeah, I've learned like one of my biggest takeaways working with, I think I've been using Claude code for three weeks, maybe four weeks. And one of my favorite things about it is to be like, okay, let's create a readme file. Okay, what did you learn from this conversation? Let's document that because Everybody who codes knows one of the hardest pieces is keeping all that context in your head and getting back up to speed. So I'm using LLMs to help solve that problem for me. Yeah, it's really powerful and it's unique. It's a unique aspect of Notebook so you can have this kind of in-context learning where you can have prose in code interleaved in this way. I find that it can really help language models, especially since you're updating it with your understanding and knowledge. You're not just like letting it, you're actually like grooming this very carefully. Yeah, that's a great way to put it. And I think that's really important to me. I gave this talk yesterday and I realized everybody thought I was just vibe coding and I was like, oh, I need to distinguish this because I don't want to send people off to just create a notebook. If I let Claude just create this on its own, like the data analysis would be wrong, especially if it was Sonnet. Like Sonnet is sloppy. Actually, Andrew Karpopi in his software 3.0 talk had a really great line that resonated with me, which was, I optimized for four line diffs, not a thousand line diffs. And that like encapsulates perfectly how I work with CloudCode because I'm really terrified of ending up with a product that I can't maintain myself. Okay, so the other thing that has changed, I now had CloudCode build me a custom annotation tool. You can see here, it follows everything you learned in this class. I got a... progress bar at the top this is my trace it's really nicely color coded and like visual for helping me understand like what section i'm in i've got an annotation box i got failure modes i've got next and back everything is key coded so i can do fast keyboard shortcuts i actually have about half a dozen uh custom annotation tools and because sometimes i'm annotating the whole trace sometimes i'm annotating a single question so i'm like loading a whole bunch of questions into the tool And just anytime I'm working on an eval and like my annotation needs change, I literally just have cloud code. Like I tell it, I like the interface from this annotation tool, but now I have new input or a new output format. Just spin up another tool. And it does it in two minutes. Like it's just so fast. So these I vibe code. I can't even tell you how this is implemented. I just test the inputs and the outputs and make sure that it's working. And here's the thing that's really exciting to me is that I started this project. trying to create a teaching tool and i was using it in my classes and like my classes our cohorts are 20 to 50 students it's not high volume something went wrong i would get an email and i would just rerun it but over the course of this project i started talking to this company vistally vistally is a software product for product discovery they have a whole flow for customer interviews and helping you synthesize your customer interviews and create opportunity solution trees and actually we're in the process of rolling out the interview coach as a beta feature inside Vistaly and what's crazy to me about this i'm going to tell you two weeks ago i had never worked in IDE two weeks ago i never had a dev environment in a prod environment and suddenly like i have to make production quality code so for the last two weeks i've been working non-stop like learning how to be a real software engineer i've been working with opus and o3 on okay walk me through error handling beat this up like where is it going to fail and i'm now pretty confident that i have a pretty good production product we are rolling out to beta customers this week we'll see how it goes but i'm crossing my fingers that i've got a real ai app that's doing pretty darn well this is amazing and as a product person designer kind of half-baked engineer it feels really good to release like a real product where i literally did every part it's very empowering and i don't i could not have done it without lms this might be the best evaled products in production that i have seen right now when someone asks me hey like what's a good example of a really like good eval like a product in production this is it there's actually there's one story that just makes me giggle when i think about it like i'm so happy that i'm at this point so a few weeks ago I ran two cohorts at continuous interviewing in July, which means I was getting way more traces than I'd ever had. And after week one, I started to see a new error. And the error was excerpts would get repeated across dimensions. So like in the opening with the story base or in setting the scene, the coach would be like, this is a great setting the scene question. And then in building the timeline, it would be the exact same excerpt. And it would be like, this is a terrible building the timeline question. That's really confusing for students. And it's one of the first times I saw an error mode that like spanned LLM calls. And so at first I was like, oh, how am I going to fix this? And my first idea was like, I need a question classifier. And then I'll only send each dimension the questions that are relevant to it. And I spent like a day and a half working on a question classifier, which sucks. Like it's so ambiguous and so hard. And I don't have lots of questions. And like, it was just getting gnarly. And I took a break and I was at dinner. I was talking to my husband about it. And I was like, oh. What if I just sent any subsequent call the list of excerpts that were already used and told it not to use them? Okay, I went from having an error mode that showed up in 81% of transcripts to having an error mode that only occurred in 3% of transcripts. And what was super fun about this is that I didn't even know how to detect the error because the excerpts weren't exactly the same. So first I had to figure out, like, how do I write an eval that even detects this error? And then I had to figure out, okay this isn't a prompt change this is actually a workflow change so how do i a b test a workflow change and then i had and it all worked and like i did it all in three days that's what i was like okay i feel like an ai engineer now oh you are definitely an ai engineer you are in the very center of the engineer there is probably you're like the example that we need to show everybody as an ai engineer to be honest I really love that you're always measuring things before you do the improvement. There's so few people out there who actually take the effort and the initiative to write the automated evaluator or whatever it is, make sure they have measured 81%, then implement the fix and then say 3%. I don't know. So much of this is because I just teach this exact same stuff in a different domain, right? In discovery, we talk about... Start with an outcome. What are you trying to achieve? If you don't know what that is and you can't measure it, don't stop. Don't start changing things. So there were so many analogies, like even Campbell's hammering on, look at your data. My equivalent of that is go talk to your customer. There's just so many analogies that I think I was really had a strong mental model to to layer all this stuff onto. There's a really good segue into this question of where do you think the product manager and engineer and whatever, how will they work together? It's very impressive, like what you've done, honestly, like to synthesize this together, you've taken a really good mindset, this whole thing. Do you think it may collapse into a single person going to be doing this going forward? Do you have an idea how you might teach product managers this, for example? Will you teach them how to code a little bit? Like, I'm just curious like what you're thinking is on this. Yeah, I wish I had a good answer. I've been really thinking hard about this. I'm a teacher by nature and so I'm starting to think about, okay, product teams are clearly going to have to do a lot of this and if they want it to be good, what does that look like? Who is doing what? And I have some like half-baked ideas because I have been thinking about it a lot, but I don't have a clear picture yet. One of my half-baked ideas is I just separated my eval code from my data analysis. And part of the reason why I did that was like to not reuse code, to be able to reuse code. But one of the benefits of that is that like now someone can dig in and do data analysis without having to see the code. And this is really important to me because I want to create like 20 of these teaching tools. I have seven different classes and this is just one of them. And I have so many ideas for teaching tools, but I can't do this iterative work on a lot of tools personally myself. So I want to engage my instructors in helping with these. And so maybe I'm the engineer, but the instructor does the analysis and the evolution. I don't know. And so one idea I have is maybe notebooks can help because I think notebooks are really good. You can obfuscate some of the code, but still let people execute it from a notebook. They can get visualizations. I think there's a new skill that we're all going to have to learn around. Okay, I wanted to create an interview coach. I had no idea how to do it on day one, but I just took a first step and I tried something. And then I got stuck and I learned how to use the LLM to get unstuck. And then I followed that pattern like 4,000 times over the last four months. And I think it's this skill of like, how do you identify the smallest piece that you need to learn? And then how do you work with the LLM to learn that? and then iterate from there because like yesterday i gave this exact same talk and someone was like okay if we were where you were four months ago how should we get started and i'm like i just told you i told you how i got started and so clearly there's like an ability there of like how do you be a self-directed learner with support from an llm and i think we're gonna have to learn how to do that even as a cross-functional team right so like The product manager has to learn some bits, but then the engineer needs to learn where, yeah, you could write an eval for this, but maybe someone with domain expertise would write a better eval for this. And what does that look like? One thing that really strikes me is your level of agency is super high. And the amount of self-help you avail yourself of proactively with an LLM is quite good. And I feel like a lot of people... fail there. Do you have sense? Is that something you can teach? Or is it... How do you teach that? I don't know. I've actually been thinking about this. Just yesterday, I started thinking about it a lot because a lot of the questions I got yesterday were around like, yeah, but what did you say to ChatGPT? I'm like, okay, forget everything you know about prompt engineering. It doesn't matter. I literally said to ChatGPT. I'm not joking. This is like where I started. I want to learn how to use a Jupyter notebook. I'm a total beginner. What do I do? Oh, I'm on a Mac. So I didn't even have a development environment set up on my Mac when I started, right? Like I was a total beginner. But the thing is like, you can ask an alum all your dumb questions because it doesn't care. And I remember when I was in college and I took computer science classes, I would get stuck and it was so frustrating. I got to go find a TA or I got to go ask another stupid smart. Stanford student that I'm going to be embarrassed asking my stupid question to. And like, I never did it. I just turned in homework that didn't work. Cause like I was too embarrassed to ask what I didn't understand. And so I was a terrible programmer in college. I was great at CS theory, which is how I graduated. But like, I was really bad at programming because I didn't know how to get unstuck. Whereas like now LLMs, like they don't, you can ask a really dumb question and they don't care. So I just, I think when you're in a conversational flow, like when you're in the web app of Cloud AI or ChatGPT.com, like prompt engineering does not matter because you can iterate, right? So you can just be like, here's my half-picked question. And then it comes back and you're like, no, I don't understand a thing you said. And I literally will copy and paste a line and be like, what do you mean by this? And so some of it is just patience to pick through it and know that you can figure it out with enough time. this past week so we're like at the 11th hour about to release the interview coach inside VistaLeaf and I was running through all my like integration tests and one of my integration tests caught an error in one of my step functions that was not caught and it didn't catch the error it just the test failed and there was an uncaught error and I was like wow I've been beating up my error handling like how do I still have an uncaught error and it was because I didn't know that this is gonna get really technical for just a second I didn't know then like JSON path errors in a step function are not catchable and they're not retryable. Like they just fail the step function. And I had written my sub step functions to return an error in the payload. And like, you can't do that. Like it has to throw an exception for the parent step function to call it. So I realized like at the 11th hour, all of my error handling between my step function to step function calls were wrong. Okay, if you didn't understand all of that, any of that, all you have to know is that This was a very embarrassing last minute thing that many realized my really reliable app was not very reliable. And this was like last Tuesday. And I panicked. And then I was like, okay, you know how to do this. Just go talk to Claude and chat GPT. And I like gave Opus in 03, like, here's my testing pattern. What are the existing patterns? What are the best practices? How do I fix it? And we talked through. three or four different error handling patterns between step functions and i was like tell me about the pros and cons of each and we picked one and then i was like okay now look at my code base and how many changes do we have to make and how do we make a project plan and i do that in planning mode like i don't want claude touching my code while we're debating how to do something and then we write a project plan that's really detailed that i've reviewed and then we literally step by step implement it and between wednesday and friday i re-architected all of the error handling in the interview coach and released it on monday morning and then and then chat gpt5 came out and i literally started asking chat gpt5 beat up my error handling and it's wow you have really robust error handling and i was like yes just so people can play with this like this visdely thing so that is it's available right now can you go sign up for this service and Okay, so part of the reason why I partnered with Vistily is that part of the story I didn't share is that when I launched this tool in my course, I was expecting students to submit their practice interviews. And they did that. But they also started submitting their real customer interviews. And this is scary because, like, I just put this stuff in Airtable. Like, I'm not really SOC. I'm certainly not SOC 2 compliant. And, like, people were sending sensitive customer data to me. And I was like, oh, I don't know how to handle that. And I started to look into it and I was like, yep, not doing that. And so what we get with the partnership with Vistaly is this is a tool where product teams are already storing their customer interviews. They are SOC 2 compliant. They follow all the data management laws. And so unfortunately, there's only two ways to try this right now. One is through our continuous, excuse me, our continuous interviewing course or through Vistaly. So if you're looking for a product discovery tool, I do strongly recommend VistaLeaf. But I am starting to think through what's a light version of the coach that I can make freely available. And actually, it's based on those very two evals. Can I at least give people feedback for free on did they ask leading questions and general questions? So I'll have more that's publicly available to at least play with probably in the next couple of weeks. That's great. And what's the best way for people to follow you or keep up with what you're doing? What do you recommend that people do? Yeah, so I blog at producttalk.org. I'm on LinkedIn and X. LinkedIn lately has been my primary social network, but I do check in on X every day. That's probably the best places. And then I'll be in the Discord. So if there's lots of questions, I'm happy. If we have time, I'm happy to take questions now. And I'm also happy to take questions in the Discord. Okay. Yeah. We could take some questions now. You have a little bit of time, Teresa. Okay. Robert has a good question. Are you thinking about capturing annotations from end users directly? Is that something that you've thought about? Or do you feel like I am the PM or I'm like building this product and I want to control all the annotation? Yeah. Robert, I, you had, we actually had a Twitter discussion about this and I forgot I was going to tackle this. One of the things I'm really excited about is. What's the customer feedback loop in all of this? Because that's my jam. And there's a few places of things in the class that I learned that I really want to experiment with. So one is you guys learn to generate synthetic data by defining like dimensions. I actually think those dimensions should come from customer needs and customer pain points. So like the recipe bot, like if you've talked to people about how they plan their meals and you know what their use cases are, like it gives you way better input for what those dimensions should be. And like. what types of variations you're going to have ahead of time. So that's a really clear area where I can see a connection between traditional discovery work and like the very beginning phases of like, how do we generate representative synthetic data? Another area is like almost every AI product has like a thumbs up, thumbs down, was this helpful? And like, I did build that into the interview coach in Vistaly. I'm still trying to figure out how to do it in our course because they get it by email. But I would love it if we could find, I want to really experiment with UX patterns of like, how can we get the customer to communicate more granular error feedback? So if they're giving me feedback on the whole response, this was helpful or not, that's great. It like, it tells me to go investigate that trace. If they give me a verbatim of why that's even more helpful, but I would love to get to the point of no, this excerpt, the tip on this excerpt is wrong. And just like what? Like ultimately our product works if it works for the customer. And so as a product manager, you can be a proxy for that. But is there a way I'm always looking for like, how do we get the customer closer to the problem? I don't, Robert, I don't have answers for you yet. Like I've been so heads down building, getting ready for this Vista Lee beta that I've just been an engineer, but I'm going to put my product management product discovery hat back on and start to run some of those experiments too. You have some background in computer science education. You did some JavaScript development early in your career. Do you think you would have been just as successful without that? Do people, people looking at this topic, should they feel like they can do what you did without that education? Yeah. Okay. So for like decades, there's been this debate in the product management world about should product managers learn how to code? i'm really glad i learned how to code in college a little bit even as a kid i coded in basic on a apple 2e but like i really learned how to code in college i'm because one of the skills i got from learning how to code is this like problem solving debugging deconstructing like a big problem into smaller parts and then also like this attention to detail that i don't think you get in anything else and so i think there's skills you learn from coding that transfer to other domains and i think coding is one of the best ways to learn those skills so there's part of me that's like everybody i think maybe coding even with vibe coding being a thing might be a proficiency like literacy i just feel like it teaches such good problem solving skills could i have done this if i didn't have that background okay so my undergraduate was symbolic systems so for people that are familiar with this at stanford it's a it's a cross-disciplinary it's a multidisciplinary program it focuses on how humans and machines process information it's also what's happening in the world with generative ai is so cool in the context of symbolic systems and i had to take classes on from computer science the analytical side of linguistics so like the underlying structure of language analytical philosophy so like mind brain philosophy and then cognitive psychology so it's really like how do we understand the information processing from four different domains? What I really took away from that program was critical thinking and problem solving. And like, ideally this is what every program is teaching. But I think when you see how four different domains solve very similar problems, you start to see this pattern. For me anyway, I started to see this pattern that I would call a more like problem solving, critical thinking pattern. And so I think I could do this. I think that experience helped me be good at these types of things. Like, I think what I was good at in this whole process was just like the critical thinking part of it. Is coding the only way to learn critical thinking? I'm sure there's plenty of people listening right now that are good critical thinkers and good problem solvers that have never written a line of code. But I think it's a really good way to learn it. And do you think, and this may be hard to predict, and I asked this before, but I just want to revisit it. If you were to teach try to teach us the product managers is your sense you may introduce some code or teach them how to code since you do think it's the one of the best you mentioned hey yeah like the best thing or would you try to go out a different way or you haven't decided yet this um this is actually a genuinely hard problem for me i'm trying to decide what i can uniquely offer so like the ai race right now is insane Like, Amon Khan is doing really cool work in his last cohort of his Maven class. He had 30 people using, like, non-technical people using cursor, which is awesome, right? Is that the right path? I don't know. I think right now it is. Is it going to be the right path in the long run? Like, I'm working out of a notebook, and I love notebooks, but notebooks are pretty technical. Am I really going to teach all my instructors how to set up a development environment on their Mac? And if they're on Windows, I can't help them. Like, they're... There's some real challenges with the way that I'm working for transferring it to other people. And like, in some ways, I wish I was still on a cross-functional team. Cause like I kind of want to partner with the engineer and play with some of these ideas. And what are those handoffs look like? One thing I am going to do, I'll give you guys all a sneak peek. I'm in the process. Okay. I'm in the process of recording some podcast episodes. I'm going to be launching a new podcast in September and I'm interviewing product. cross-functional product teams about how they're building their AI products. So it's not the like how I AI Clairvaux podcast and how people are using AI in their day-to-day workflows. It's here's the story. It's what I did today. Here's the story of an AI product. Here's how it came to fruition. And what I'm really interested in covering in those stories is who did what and how do we collaborate? Because I think this is a really open question and I don't, I'm not sure what to teach yet. But I do want to help. Like I do, as an educator, I will be doing something in this space. I'm just still figuring out what the right something is. I would love to see a visual map of how you went about annotations to the app that you're building. Specifically, there's probably a lot of micro decisions that you had to make along the way. And so could you unravel some of those? Were they difficult to make? how did you know whether something was like a micro decision or something that like required more like you learning something new or figuring out something new yeah i can talk about some of the kind of key decision points i made and how they came about and maybe that'll illustrate this a little bit yeah like one of the big ones was i started the interview coach was one prompt like it was this monster prompt and the problem i saw with that was that in the coaching responses like one of my biggest failure modes was the coach would give feedback on one dimension, but use criteria from a different dimension. So it was confusing the rules across dimensions. And I think like around that time I learned about workflows and I was like, oh, maybe this needs to be a series of LLM calls. And so I broke it up into four LLM calls, but then that introduced some new problems. Like now, like setting this it's not too much setting the scene like setting the scene doesn't know about building the timeline and it's literally evaluating the entire interview based on setting the scene and so like that that change had a really big impact like my overall quality went up when i split up my llm calls but my introduced new error modes and this is something that like i think is probably really common there isn't a magical change that makes everything better it's like a change can improve some things and then maybe introduce some new errors and i i then started once i had four lm calls then i really got stuck on i even asked this in the discord do i do evals for each lm call or do i do evals for the whole response and i actually started by doing them one dimension at a time and it got really overwhelming because i was just like i'm going to be doing evals for the rest of my life and i realized like by doing it one dimension at a time i was like finding a lot of errors but a lot of them weren't critical errors and so then i was like oh maybe i actually should do evals on the whole trace i should do analysis annotations on the whole trace because then i can still find dimension specific errors but i'm prioritizing all of them against each other and that actually made a really big difference and i still do individual dimension annotations when i'm working on an eval that's specific to that dimension But I'm doing my error analysis and like failure mode prioritization on the whole trace because that's like what's impacting my customer. And that was a pattern I really struggled with. What's the right way to do this? And I've tried both. I've spent a ton of time annotating things I didn't need to annotate as a result. And I just, I think for me, my conclusion is I should start with the customer experience. The customer gets the full trace. I'm going to look at the whole trace and it helps me prioritize a little bit better. So that was a key decision point. Choosing to move from Airtable interfaces to my own annotation tool. Like I know in class we learned build your own annotation tool and I was like I did an Airtable. But like really building my own annotation tool like I still I'll be annotating I'll be like four traces in and I'll be like like even this morning I was annotating actually right before this and I was like Why could I add like little pills above my failure mode? So like in this set, once I've used an error mode, it should just be a pill. So I don't have to go back and remember what the stupid failure mode was. So like I'm constantly thinking of ways to make annotating faster. And I literally just stop what I'm doing. And I'm like, Claude, can you add this to this annotation tool? And I'm forcing myself to do that because like you learn in this class, like annotation is so important. And if it's not effortless and fast, Like you're just not going to do enough of it. And my annotations are hard. Like that coach response is a lot of text. So like my annotations are slow. When I'm doing full trace annotations, I have other annotations that like, like when I was doing my question classifier, I actually did my annotations with Claude. I gave Claude a list of questions and I just said, ask me like Claude code. I said, just show me one at a time and let me just tell you the failure mode. And we just ran through like a hundred traces in like less than 10 minutes. and i was like whoa that was a cool annotation interface that i just used my l1 for um so i think like always just thinking about what's the right tool for the moment and then i think the other big thing i've learned is just it sounds so obvious but take breaks step away because i've definitely had moments where like i'm banging my head at something and i go and have a meal or i go for a walk and i come back and just a way simpler solution comes to mind And that's like a general problem solving tip, but I, even though I know it, I forget it often. I'm like, I literally am having more fun than I think I've had in my entire career. And I'm obsessed right now. And I code all day long. My husband jokes that I broke up with him for Claude. And so like, I really have to rein in. Like I have to, I now set a timer on my watch to remind me, just get up and step away. It's just so fun. It feels like the 90s. It feels like the beginning of the internet. And everybody's like geeking out and trying to learn this stuff and everybody's sharing with each other. It just, it feels really similar and really fun. Yeah, the amount you're having fun is definitely coming through the screen. I can definitely feel it all the way over here. Level of energy, really good. We can keep going on forever, but let's wrap things up. Yeah. And then for folks like in the class, if you guys are on a cross-functional product team and you're building a product, like an AI product, reach out because I'm definitely looking for teams to interview. And you don't have to be an expert. Partly what I want to show with this podcast is how we're all figuring it out together. Right? I'm not, I've done the guru thing. I'm over it. Now I'm just into the let's be a community and share with each other. So that's my goal. It's just how do we share more stories? That's great. that's really good humility i would say i'm really impressed i would still call you a guru like yeah thank you so much thank you both i really enjoyed this class and clearly learned a lot oh definitely you definitely learned a lot this is yeah a plus thank you so much thanks

Join the AI Evals September 2026 cohort: https://maven.com/parlance-labs/evals?promoCode=yt-2026

Teresa Torres has trained over 70,000 product managers on how to run good product discovery. In this session she walks through exactly how she went from vibe checking her AI interview coach to a fully automated eval system in four days, using Python she'd never written before and tools she'd never heard of.

Timestamps:
00:00 Intro: Teresa Torres, product management legend and first cohort student
01:50 The product: an AI interview coach for customer interview practice
07:00 Why she got stuck before finding this course
09:04 First Sunday night: writing Python she'd never used before, with ChatGPT as a guide
10:30 Using Airtable as her first annotation tool
12:05 Seeing her failure modes for the first time: "holy crap, my coach sucks"
15:00 Her first two evals: an LLM judge and a code assertion
19:12 The insight an engineer without domain expertise would have missed
24:58 Fast feedback loop: how evals changed how she develops the product
31:09 Today's setup: VS Code, Claude Code, custom annotation tools
40:00 The A/B test that took an error from 81% to 3% of transcripts
46:15 How product managers and engineers should split eval work going forward
58:38 What background do you actually need to do what Teresa did?