The #1 Skill PMs Need in 2025: AI Product Discovery Masterclass by World’s Leading Authority

What does discovery look like in the age of AI? Does it change everything or does it change nothing? You know, I've been getting asked a lot, like when delivery is free, do we still need to do discovery? And I actually think when delivery is free, discovery becomes more important. In today's episode, I sat down with Teresa Torres, the legendary author of Continuous Discovery Habits. This is a book that I myself have read multiple times, marked up multiple times. So many PMs are doing customer interviews, yet they're products and their features fail. Why? She has worked with over 17,000 PMs across the world in over 100 countries. So she brings the insight you need to improve your discovery, not just for regular features with AI, but for AI features. Here's the challenge I see with prompt engineering. We all have experience like chatting with ChatGPT or Claude, and we're in a conversation. If we get that first prompt wrong, we can immediately refine it. But when you're building a product, the prompt can't be refined by you. Once it's live in your- product. There's no refinement. It's a one shot. So what are the signs that PMs are doing fake discovery? Nothing in their backlog changes. They don't kill any ideas. There's a lot of discovery theater out there. Before you go, Teresa, I have to ask, how big is the business of Teresa Torres? Yeah, so I'm a c***er. Teresa, welcome to the podcast. Thanks for having me. I'm excited to do this. As I was saying off air, you are on my S tier of guests along with Marty Kagan. You are the two guests I wanted most when I dreamed of starting this podcast. And that's because... I think you have probably advised more PMs on Discovery than anyone else in the world. What would you say the number is at now? Yeah, it kind of depends on how we count. So when I was coaching teams directly, I would work with about 30 teams a year. I did that for over a decade, so probably over 300 teams. And what that means is like weekly calls for multiple months. So I was sort of in-depth. Through the Product Talk Academy, we have over 17,000 students, which is pretty mind-blowing. And they come from... from over 100 countries. Wow, 100 countries. I didn't even know PM was practiced in 100 countries. So you have really seen the whole world of product discovery. And it's interesting because so many PMs are doing customer interviews, yet their products and their features fail. Why? Yeah, this is a complicated topic. I think there's a lot of reasons for this. I think the primary reason is that we're not that good at interviewing. So a lot of teams that go into interviews with the intent of exploring their solution and getting feedback on their solution. That's not really the best way to get feedback on our solutions. Our goal in our customer interviews should be to learn about our customers. Even if we know that's the goal of the interviews and we don't talk about our solutions, we tend to ask really unreliable questions like, what do you like and dislike about different things? Tell me about your experience broadly. And so one of the things that I teach, I introduce this idea in the book, we teach it through all of our programs, is this idea of story-based interviewing. So how do I talk to you and collect a reliable story about your past experience? So I learned about what you actually do and not what you think you do, not what you aspire to do, but like in reality, what did you do recently? So that I can make sure that I'm building a product that fits in your lived world. So it sounds like people are asking too many hypothetical questions. Here's a prototype. Would you like to use this? What would be the better way for them to approach that conversation? Yeah. So like, let's look at this in tiers. So the first is a lot of people present a solution and say, would you use this? That's terrible. Unreliable feedback. We're not good at predicting our future behavior. Also humans want to be nice. So we're going to say, yeah, of course I would use that. And even if we think we're being honest, it's actually, we're optimistic about our future time and about what we might do in the future. So we might actually genuinely think we're going to use it, but it doesn't mean we are. There's actually better ways to test our solutions. So I really like assumption testing when we're evaluating solutions, which is a whole different activity from interviewing. We can get into that if you want. What I like to use interviews for is let me just learn about you. And so what I want to do sort of the next level, like people learn, okay, I should ask an open-ended question. So they'll be like, tell me about your experience with my product. The challenge with that type of question, it is open-ended. I might learn a lot about you, but what I'm going to learn is what you think you do, not necessarily what you actually do. And so to fix that, I want to ask you, tell me about the last time you used the product or even better, tell me about the last time. you solved the problem the product was designed to solve. That is exactly what I've learned through hard trial and error and reading and rereading this book. It actually works, folks. And you briefly mentioned assumption testing. Can you give us the 30 second overview of what that is? Yeah. So it's this idea of we tend to fall into the trap of big idea testing, which means we have to do all the design up front and then usability test it or we have to build it and A-B test it. The challenge with those strategies, they're great to have in our toolbox. We want to do them eventually, but when we're doing them in discovery, we're learning after we did all the work, if the idea would work or not. I prefer to learn if something's going to work or not before I do all the work. And so the key to that is to break the idea down into its underlying assumptions. So what needs to be true in order for this idea to work and then to test those particulars. And we can tend to test assumptions much quicker. We don't have to do all the design work. We certainly don't have to do all the engineering. work and we can start to collect data on whether whether an idea would work or not before we do all the work. Okay, and I think this comes together, although you can correct me if I'm wrong, in the continuous discovery habits system. Can you break down what exactly is continuous discovery? Yeah, so this framework I developed to help newly empowered product teams figure out what in the world to do. So let's talk about that for a second. I think historically, product teams have been asked to deliver specific features, usually by specific dates. We call these roadmaps. Sometimes teams are creating those roadmaps themselves and they've had to deal a little bit with what do I put on my roadmap. But for a lot of product managers, their stakeholders are putting things on their roadmap and their job is literally to just deliver. And so what I saw happen is companies started to shift from an output focus to outcome focus. So now they're saying, okay, teams, we get it. The future is uncertain. We don't know what you should build, but you need to reduce churn or you need to increase retention or you need to drive engagement. And these teams are like, what? You've always told me what to build. I don't know how to do this. And so the continuous discovery habits are about how do we answer these evergreen wide open challenges like engagement and retention or even customer acquisition. And so it starts with having a clear idea of what your outcome is. That's typically a metric. It can be something like increase engagement. Usually it's more specific than that. We're defining the types of engagement we want, but that's good enough for now. And then the habit on the left here is we're interviewing. And we're interviewing week over week with the goal of understanding our customers. So we're not testing our solutions. We're not evaluating our solutions in our interviews. We're really trying to understand who are we building for. We're setting up a good building with cadence. And then the visual in the middle, I should have started there, is an opportunity solution tree. This is a visual that I designed to help people keep track of their messy discovery work. So it starts with an outcome at the top. We're interviewing to uncover the opportunity space. Opportunities are unmet customer needs. pain points and desires. So as we interview, as we learn about what people did in the past, we're going to uncover friction. We're going to capture all that in the opportunity space. Eventually, hopefully not after too long, we're going to choose a target opportunity. I really am an advocate of comparing contrast decisions when evaluating solutions. So we're looking at multiple solutions for the same target opportunity. And then we're using assumption testing to break those solutions down into their underlying assumptions. And then assumption testing to evaluate which one looks like a winner. So this diagram is timeless. I still refer people back to the core continuous discovery loop. But how has AI changed continuous discovery? I think it depends on who you ask. And I really am conflicted about this. So let's talk about two different paths of how AI impacts this. There's the path of how does AI affect how I do my day-to-day job. And then there's the path of how does my job change when I'm building AI products. So in the first path, I love AI as a thought partner. So like even if I'm defining outcomes, I might be chatting with Claude or ChatGPT about my outcome and how I might better frame it or how I could better measure it. It's probably not telling me what my outcome is because it needs a lot of business context for that. I mean, you know, now people set up projects with all their business context and maybe it could help with that. But typically our outcomes are coming from our executives. And so maybe it's... we're using it to refine that activity. I do know some teams that are starting to experiment with having AI interview their customers for you. I actually think this, we live in a world where this is very possible. I'm a little bit concerned about what it says to our customers. Like we really want to learn about you, but not enough to spend our own time doing it. I also have some concern about like one of the benefits of interviewing beyond what we learn from our customers. Just this act of having firsthand exposure to our customers helps us build empathy for them. It helps us see the gap in how we think and how they think. And I don't think like reading a transcript is going to get you that benefit. So I'm still a big advocate of humans talking to humans. One area where I'm really torn on is on synthesis. So I know a lot of teams are starting to use AI for synthesis. Here's what I like about this. I know a lot of teams that do no synthesis. They conduct interviews, the notes go into a folder, they never look at them again. If that's what you're doing, I think AI can help. My caution... is I've experimented a lot with like my synthesis versus like Claude's synthesis or ChatGPT synthesis and it misses a lot. You have to work really hard to get it specific to help it really identify opportunities. I've been running these experiments. I will probably release tools in this space eventually. It's like 60 to 80% good. And I worry about what we lose in that 20 to 40%. I also worry about what we lose when humans aren't in the data. I think our brains change when we spend that much time in our data. So I'm not going to say never on that synthesis part. In fact, the way I'm starting to think about it is for our everyday mundane things that we work on, maybe we don't need to go deep. And AI synthesis is more than adequate for our differentiators, for the things that really differentiators from our competitors in the market, I'm probably going to go deep and still do my human synthesis. And then I know David Bland is experimenting a ton with AI in helping you identify assumptions and even helping you identify assumption tests. And I think there's a ton of potential there too. I just, what I fall back on is I'm a big believer in AI augmenting human work, not replacing human work. And so I really caution people that when they start to use AI in discovery to really think about how does it accelerate, but not replace what you're doing. Yeah, that's what I keep advising people. Everybody keeps asking me, Akash, should I be using AI to synthesize and report out on my discovery? And I feel like what it's doing is it's preventing you from improving your own brain as an LLM, right? We keep wanting to load our brain up with the right context. And the magic of continuous discovery was that you were loading it up with this beautiful context from your users of actually talking to your users. And if you're outsourcing all of that. especially the interviewing part, you might lose some of that context building. Yeah, I know that like especially interview synthesis is so time consuming and it takes a lot of work. I'm actually really interested in this as a problem space of like how can AI speed up that work? What I'm wary of is using AI to completely automate that work. And there's a good reason for this, right? Like we all have access to the same AI tools. Like what's eventually going to be your differentiator if we're just all outsourcing it to the same tools. Now you could argue if you're conducting better interviews and you have better inputs, yes, that is a differentiator. But I also think there's something that we can, that humans will do. Not just that humans will do it better than AI. but how humans change when we do the work. That's the part I'm most worried about losing. So what I'm interested in is like, what are the tools that help us do it better and faster, but still enable us to get our hands dirty and to do the work. Yeah, maybe I can do 60 to 80% of it, but let's make sure that we then go in on top and do some of the work, do some of the context building to enhance it and also to create differentiated insights. Yeah, absolutely. So that's like the AI, how does it affect our workflow? Do you want to get into like, how does discovery change when we're building AI products? I want to get there in just a second, but I want to stay on the workflow one more second because there's one other element of AI I want to talk about, which is AI prototyping. And some podcast guests have called it the golden age of the feature factory because executives, you know, they can just come up with a prototype. They can send it your way and they can say, hey, can you please do a couple of user tests and then ship this into production next week? And that's actually a reality for PMs I'm talking to these days. So how do you. correctly pull AI prototyping into these discovery workflows? Yeah. Okay. So I'm going to start by, I am a huge fan of AI prototyping. It really feels like magic. I actually just interviewed 17 product people about how they're using lovable at work and we'll be publishing this blog post on August 13th. So a huge deep dive on what are people doing at work with AI prototyping? You know, I've been getting asked a lot, like when delivery is free, do we still need to do discovery? And I actually think when delivery is free, discovery becomes more important because if we're building anything and everything and just shoving it into our product, we are going to drive our customers nuts. There's going to be no product coherence. They're going to have change fatigue. We're going to have feature bloat. And what comes to mind is, have you seen the Simpsons episode where Homer got to design his own car? Oh, I have to hear this story. I haven't. Yeah, this is the best analogy I have for this. So I forget the details because I saw this probably 20, 30 years ago. But Homer Simpson got to design his own car. And you can imagine it's Homer Simpson, right? So his chair is like a big recliner and he's got cup holders everywhere for his beers and he's got donut holders. And like, it's just a ridiculous car. It looks silly. It probably doesn't even drive. Like he's just thinking about all the things that like he thinks matter, but actually don't matter because actually getting from point A to point B is what matters most in the car. are. And I actually think this is the biggest, the best analogy for what might happen if we just let anybody tack on features to our product. So I think like what I love about AI prototyping, we can now assumption test so quickly, like literally at the push of a button. We can do much higher fidelity assumption testing really quickly, but I don't think it changes our discovery work other than it speeds up a really important step. I think we still want to be really considerate about what do we put in our backlog and what are we actually releasing to customers? Because the biggest impact of change on our cost is on our customers. It's not on our engineers. Like sure, we used to maybe do discovery to like save our engineers time from building the wrong thing. But the real goal of discovery was to not exhaust our customers with the wrong features and constant change. Take me through, I guess, the life cycle, right? Typically, we would do, we would like explore the problem space. We would come up with our assumptions. We would test our assumptions. Then finally, the designer might, it might be worth their time to create a prototype. Then we'd put the prototype in front of them. Nowadays, people are just jumping straight to the prototyping. What is the actual life cycle it should be? When should you bring the AI prototyping into the process and how exactly should you use? Today's episode is brought to you by Miro. Let me ask you something. How many tools are you juggling just to get a single project across the finish line? One for brainstorming, another for planning, something else for tracking tickets. That's where Miro comes in. It becomes an all-in-one collaboration workspace. Whether you're consolidating user research from several interviews, developing and synthesizing product briefs or a wireframe, or project managing development, Miro brings everyone into the same space. It's fast, intuitive, and fully loaded with features like project templates, two-way Jira Sync, and integration with software like draw.io and plant.uml. Miro's AI features can be used to synthesize elements in a board to develop a ready-to-review product requirements document in seconds. If you're tired of tab overload and scattered workflows, try Miro. Head to Miro.com and see why over 90 million users choose Miro to guide from idea to outcome. Today's episode is brought to you by Jira Product Discovery. If you're like most product managers, you're probably in Jira, tracking tickets and managing the backlog. But what about everything that happens before delivery? Jira product discovery helps you move your discovery, prioritization, and even roadmapping work out of spreadsheets and into a purpose-built tool designed for product teams. Capture insights. prioritize what matters, and create roadmaps you can easily tailor for any audience. And because it's built to work with Jira, everything stays connected from idea to delivery. Used by product teams at Canva, Deliveroo, and even The Economist, check out why and try it for free today at atlassian.com slash product dash discovery. That's A-T-L-A-S-S-I-A-N dot com slash product dash discovery. Jira Product Discovery. Build the right thing. Yeah, I like it in the context of assumption testing. So what does that mean? It means I already have an outcome. I've already conducted, let's say, three to four interviews. I have a draft of my opportunity space. It's going to continue to evolve as I keep interviewing. I've chosen a target opportunity. Ideally, I've brainstormed multiple solutions for that target opportunity. So we're setting up a good compare and contrast decision. Now with AI prototyping, you might say, Teresa, I can actually prototype all three in a day. So why wouldn't I just do that? Here's the challenge with whole idea testing. Okay, great. We just made it free basically, almost free, to prototype three ideas. So I've got three high fidelity interactive prototypes. Why can't I just go test those? First of all, testing a whole idea with a customer takes a long time. It's a big usability test. There's like, and by a long time, it might be 20 minutes per customer, but that's a long time. And I'll explain why in a second. So we're already fatiguing our customers because we're going through 20 minute sessions. We're doing a lot of them. And then what do we get in that feedback session? We're getting like random bits of feedback on different parts of the idea based on what resonated or didn't resonate with that customer. It's not very structured. If instead I take my ideas and I break them down into underlying assumptions. So let's say that I'll give you a recent example from something I'm building. I'm building an interview coach. It gives my students feedback on how they conducted an interview. When I was like building out this solution, I had some like technical challenges. So my students conduct interviews in a Zoom breakout room during class. Zoom doesn't let me record every breakout room. That's already a technical challenge. I need recordings of the breakout room. I need to go from recording to transcript. And then I need to go from transcript to the coach. So when I'm like testing this idea, will it work? I have a number of steps that I have to test. Will students like stay in the main room long enough for me to give them permission to record? When they go to their breakout room, will they remember to record? Will they be able to find the recording on their computer? Will they be willing? to submit it to my extract an SRT file tool. Will they then copy? Like it's, there's a lot of friction in this process. It's kind of a nightmare. Thank you, Zoom. And so when I'm testing this idea, I don't just release it out there and hope they do all the things that they do because I don't get visibility into what step broke down. But I can test each individual assumption. So I can first like with one group just be like, okay, we're going to record our practice sessions today. Here's how it's going to work. And I'm going to see like, did I have to go pull people out of their breakout rooms to get them back to the main room? Because that's the only place I can give them permission. Right? And so like I'm methodically testing each of these individual pieces. And so it tells me exactly where the breakdown is and exactly what I might need to iterate on. Versus if I just released a product. and students didn't actually submit a transcript, I would have no idea where in that process it failed, especially because I'm using third party tools. I can't instrument Zoom. Even though AI prototyping now makes it really easy to create a prototype, I still want to create a prototype for a particular so I know exactly where in my process the breakdown happened. So I'm testing each particular in isolation. Okay, so to recap that back to see if I got it right, where we want to use AI prototypes is after we already have identified the opportunity space, after we've brainstormed several solutions, and we don't want to just prototype the whole solution. We want to prototype a particular element of that solution in order to assumption test around that element of the solution. Exactly. And what are people doing wrong with AI prototyping tools? What do you most often see them doing? You know, I love, well, the first thing is a lot of people are just starting with solution ideas. They don't have any idea what customer problem they're solving. And this is really rampant across all product management, right? A lot of us, like we think of it as like shiny object syndrome. My job is to have a creative idea. And this I think is a really big misunderstanding of product management. Like our job is to solve customer needs. And so we should always be working in the context of a customer problem, a customer pain point, a customer desire that we're working to address. And I think AI prototyping makes it even easier to be shiny object, to fall prey to shiny object syndrome, because I can literally just type like, I had this half-baked idea and now you can make it real. And I think it takes a little bit of discipline to be like, no, hold on, what are we doing this for? So staying a little bit more on these fundamentals of continuous discovery for any feature. What are the most common bad interview questions that PMs ask outside of the hypotheticals? What are the common mistakes that your interview coach or you when you're coaching people are fixing for them? Yeah, so let's assume we're conducting a story-based interview. So people know already to collect a story. So they're starting with something like, tell me about the last time you used lovable. What's hard is that people aren't good at telling their stories naturally. And so people underestimate how much work goes into the skill of, we use the verb excavating the story. So I love this verb because it's a really good analogy, right? Like typically we excavate artifacts with like small hand tools and like gently pull out the artifact from like a dig site. This is actually a great analogy for interviewing because in conversation there's this like 50-50 norm. I say something, you say something, I say something, you say something. In an interview, we actually want to break that norm. We want the participant doing most of the talking. And so we have to communicate, no, we really want all the detail, right? So if I say like, tell me about the last time you watched Netflix, you're gonna be like, I watched a movie last night after dinner. And I have to actually situate you in that moment. Like, oh, it was after dinner. Okay, so let's put you back in that moment when you're finishing dinner. Tell me about the decision to decide to watch something. and then walk me through the whole narrative of like, you went from the dining room table to the living room and turned on the TV and what device did you use? And like, I'm going to pull out the whole narrative. Whereas what the biggest mistake people make is they start with the story based question, but because they don't know how to excavate the story, they don't know how to pull out that narrative, they start guessing what happened. And so their questions are guesses of like, oh, did something go wrong while you were watching? Or Did you look at recommendations, right? Instead of just using like temporal prompts, what did you do first? What came next? So there's sort of this skill of digging out a story that I think a lot of people underestimate. And once they're done with an interview, what is the right way to capture the insights from that particular interview? Yeah, okay. So... I want to clarify, I teach one form of interviewing, story-based interviewing. The reason why I teach this is while it is a skill and it does require practice, the idea is simple so we can all grok it, right? It's also, I think, the best form of interviewing for continuous interviews. For a product team that is interviewing a customer every week, it's allowing us to continuously invest in our understanding of customers. Now, the reason why I had to caveat that is when we start talking about how do we synthesize this interview, The goal of this synthesis is supporting week over week product work. So this is different from like interviews that your user research team might conduct or how they might synthesize a dozen interviews at once. So I want to just clarify that. But in this context, if I'm on a product team, I'm doing an interview every week. What I want to be doing is I want to be capturing what I'm learning from each interview. So I distinguish between single interview synthesis. and then a cross interview synthesis. So this interview snapshot that we're looking at is a it's just a one page template and it's designed to help you synthesize a single interview. And we're doing a few things here. We're starting with the biggest thing here is at the bottom. It's actually this experience map. So if I'm collecting a story, I want to identify what are the key moments in that story and how do I capture them in an experience map. This is going to help me remember the story over time. It's going to allow me at a glance to look at this snapshot and be like, oh yeah, I remember that story. It's also going to help me find patterns across stories and it's going to help me give structure to the opportunity space on an opportunity solution tree. I'd say the next most important thing on this snapshot is the list of opportunities. This is what's making our interviews actionable. So what opportunities, what unmet customer needs, pain points and desires did I hear in this interview? and I'm capturing as many as I can. And then the rest of the elements here are just helping with like the quick facts are sort of my customer segment data. So how do I put this specific story in the right context? Is this an engaged customer? Is it a new customer? Are they a large enterprise? Are they a small business? These quick facts are going to vary business to business, but you're settling on half a dozen, maybe up to a dozen kind of quick facts that help you situate the story in a specific context. The photo and the quote again are memory aids. So I like to pull out a salient quote that just reminds me what did I learn in this interview? This is kind of a memory trick. Like I will tell you, I still remember interviews I did years ago because of the salient quote. So it's just another way to quickly look at the interview snapshot and be like, oh yeah, I remember that person. And then of course the insights is just other notes that we've captured that we're not, they're not quite actionable yet. We're not sure where they go, but we don't want to lose them. So this is for a single interview. How does it look for the other scenario? Yeah, when we're synthesizing across interviews, this is where I like to do this in the context of my opportunity solution tree. So I'm starting with my outcome. I'm looking across my interview snapshots, which not all my opportunities for my snapshot go on my tree. My outcome acts as a filter. So which of these opportunities do I think have the potential to drive the outcome? And those are the opportunities that I'm moving over to my tree. And then I'm looking, I'm using the experience maps to give structure to the opportunity space. So what I teach teams is once you have three or four interviews done, you can start to look at your experience maps for the individual stories and then what you're looking for is the experience map that encompasses all of the stories. So it's like a super experience map and then the moments in that super experience map map to your top level opportunities. And this is really important because it guarantees that each branch in your tree is distinct because each branch represents a different moment in a story. And so the needs and the pain points that show up will be specific to that moment. This helps to reduce dependencies between our opportunities and allows us to work on one opportunity at a time. You're speaking to a part that I think is very nuanced and is part of the skill that people are going to develop as they do more continuous discovery, which is how to update the opportunity solution tree. Can you go in a little bit more depth for us? How are we adding things here? How are we crossing them off? What is the typical cadence looking like to refine this OST? Yeah, so the first thing I'll highlight is this really is a living document. It's not a one-time activity. It does have prerequisites. You need an outcome. I recommend you start with at least three to four story-based customer interviews with snapshots for those interviews. So you've already done your experience mapping. You've already identified your opportunities. And then from there, you can start to work on your first draft of the opportunity space. And I cannot emphasize enough, it is a first draft. And so again, we're finding those top level opportunities based on the key moments across our stories. We can then take all the individual opportunities we're hearing that are related to our outcome and just group them under the moment in which they occurred. So we can say for this opportunity, what moment in the story did it emerge? And then for each branch, we're going to start to look for relationships between the opportunities. Which ones are children? Which ones are parents? We probably have to add some parents to give structure to the opportunity space. Let's talk about an ideal timeline. Like if I'm starting a quarter with a new outcome, in week one, I'm probably front-loading three to four interviews. So by the end of week one, I have a draft of my opportunity space. That sets me up in week two. I can choose a target opportunity. I can start brainstorming solutions. I can start assumption testing as soon as week two. I'm going to continue to interview every week. And every three to four interviews, I'm going to revisit my opportunity space. So if I'm working on a quarterly basis, if I'm doing one interview a week, then I'm roughly revising my opportunity space every three to four weeks. Beautiful. I don't think enough people are grokking that. It's an iterative document and it's also a communication tool for your stakeholders. Absolutely. So what are the signs that PMs are doing fake discovery? Nothing in their backlog changes. They don't kill any ideas. They always end up building what they started with. They're not considering multiple solutions. I mean, there's a lot of discovery theater out there, which some of this is just lack of know-how. I don't think teams are like, like... intentionally putting up a facade. I've actually never seen a team that's like like cheating the system. I think it's more like there's like personal work we have to do to make this work. Like we have to be able to set aside our ego a little bit. We have to distance ourselves from our favorite idea. We have to come and do an interview with curiosity, especially when it's the seventh interview on the same topic. We have to recognize that each customer is unique and actively dig for what's unique about this customer. And I think plenty of product managers and everybody else on the product trio aren't getting the coaching they need or the support they need to do that work. Plenty of organizational context don't give them the space to do that. We still reward people for being right and for having strong opinions. And so we're not rewarding. them for being curious and we're not rewarding failures that indicate we actually learned something new. So I don't put this entirely on product managers. I think a lot of our organizational context doesn't support this way of working. But I do think individuals can make more progress than they think they can, even if their organization doesn't support it. Is your AI product burning budget without results? Drowning in frameworks but can't improve your LLM performance? That's exactly why Parlance Labs exists, today's podcast partner. They're an engineer-only AI consulting firm led by Hamal Hussein. No slide decks, just practical solutions. They've helped 30-plus companies like Honeycomb, DBT Labs, and even Langchain dramatically improve their AI through systematic evaluation. Here's what makes them different. They don't want dependency. They teach your team to evaluate AI systems and optimize performance so you can fire them when ready. Ready to stop guessing and start improving your AI? Visit parlance-labs.com. That's p-a-r-l-a-n-c-e-l-a-b-s.com. Or email consulting at parlance-labs.com. Before we dive deeper, let's talk about something every PM faces, getting alignment on product decisions. You know that feeling when you're trying to explain a user flow to engineering or justify a design choice to leadership and you're just describing it with your hands? That's where Mobbin comes in. Mobin is the world's largest library of real-world mobile and web app designs from industry-leading apps like Airbnb, Uber, and Pinterest. Instead of spending hours taking screenshots or hunting for inspiration, you can instantly find exactly how successful products handle onboarding, paywalls, checkout flows, whatever you're facing. Over 1.7 million product builders use Mobin to benchmark against best-in-class products and show their team's proven solutions. Whether you need to convince stakeholders there's a better way to handle user activation or research a top app's approach for future discovery, Mobbin gives you the visual proof to back up your product decisions. Check out mobbin.com slash akash, that's m-o-b-b-i-n dot com slash a-a-k-a-s-h, and get 20% off your first year. If you're not talking to customers weekly, are you really doing real product management? Yeah, in the past I've said no. And I've been very black and white about this, but I've softened my answer. And I think it's because there's a lot of toxic messages in our industry right now, where if you're not doing all the right things, you're not a real product manager. Here's my answer now. If you're doing what your company expects you to do, you are a product manager. Like fundamentally, that's our floor. Like your job is to do what your company expects you to do. Period. Can we do more than that? I think so. And I really think everybody has more agency than they realize. And so I want to encourage people to like step into this, even if they're not getting organizational support for it. But I don't want to tell them what they're doing is wrong. I feel like something has changed recently. I feel like it's everybody's building their personal brand and like LinkedIn has become this toxic dump of everybody telling you you're doing your job wrong. And I really don't want to contribute to that. I think we could All of us have room for improvement and there's more that we could be doing. But fundamentally, our job is to do what our company expects us to do. I love that message. There's way too many people telling you if you want to be a great PM versus good PM, I think it's like, yes, Ben Horowitz wrote a great piece of content 25 years ago and it resonated then. But can we please move on from that messaging? So I want to touch on the other side now of AI and product discovery. What does product discovery look like for an AI feature? Yeah, I'm just learning about this myself. So I'll share that, let's see, it's late July. I am about four months into building my first AI product. And it's completely opened my eyes to just how different product management is for an AI product. So let's first start with the differences of like, what's different? And then we can get into like, where does discovery fit in there? So I think there's a few things that are different. And I think what's hard for me about this topic is I'm still struggling to separate what's product work and what's engineering work. And I really think AI products are going to blur our roles a lot more than they have in the past, which I'm actually really excited about because I love it when our roles blur. I've always been a boundary spanner. And so I think I want to see more of that, but like, let's talk about this. I'm actually working on a blog post where I'm just doing this for my own sense making, trying to understand what are the big components of an AI product that are different from a deterministic product or feature. So the first one is, I know people are starting to use this term context engineering, which I like a little bit better than prompt engineering. Here's the challenge I see with prompt engineering. We all have experience chatting with ChatGPT or Claude, and we're in a conversation. If we get that first prompt wrong, we can immediately refine. And so when we think prompt engineering, we're like, oh, we know how to do that because we talk to these tools all day every day. But when you're building a product, the prompt can't be refined by you in the, I mean, it can be as you're developing, but like once it's live in your product, there's no refinement. It's a one shot. So that prompt has to work. So there's this skill of like, how do we write prompts? How do we instruct an LLM to reliably do the same thing? over thousands and thousands of instances and I think people underestimate how hard this is and this is a very different problem than like I'm just chatting with chat GPT. So I think that's one piece and we can get into that and like what how discovery can inform that. I think a second piece is always in product work we have this task of decomposition. We have this vision of this big feature and like how are we going to get there over time but I think with AI products the decomp... composition task is twofold. How are we going to get there over time? But also how are we orchestrating, like how do we break up the tasks that LLMs can be good at it? So like I went through this on my own learning journey. My interview coach started as one prompt and now it's seven prompts and it's quickly growing. And so then it introduces like this sort of orchestration question, which is a little bit engineering, but I think it's also product. And this is where we get into things like which tools should we use? Are we using MCP servers? Are we using RAG? Like there's sort of this messy, I'm just going to call it orchestration. And then there's a third piece of like observability. Are we collecting traces? Are we doing that in an ethical data practice way? Are we informing our customer? But we have to store traces. And for those that don't know, a trace is just a LLM prompt plus the response. So all the back and forth between the LLM and the end user. And then I think the last piece of this is like, how are we evaluating quality? And this is where evals come in. And so I think broadly, and then there's the ongoing like maintenance of a feature, which I think looks very different than a deterministic feature. So it's kind of the five buckets that I've kind of started to noodle on of this framework of how our AI products different. And I think discovery can inform each of those, but let me pause there before I get into that. So let me recap. Context engineering, orchestration, observability, quality, and maintenance. Did I get that right? Yep. So let's start with a little bit of a deeper dive on context engineering. How does that change and how should we Discovery be playing into that? Yeah, so I was really surprised when I, so, but just to give a little bit of context, the AI product that I'm in building is an interview coach. So in our courses, people practice interviews. they submit their transcripts and an AI coach gives them really detailed feedback. Like it pulls out excerpts, it gives them coaching tips. And when I started building this, when I first started experimenting, I was literally in a cloud project. I uploaded a lot of my course content. We already had a rubric in the course that we gave to students so they could give feedback on each other's interviews. And I had this really big insight that context engineering in the context of an LLM product is very similar to teaching humans a skill, right? So to teach an LLM to do a thing is very similar to teach a human how to do a thing. It's really about what context do you give them and when so they're able to do the skill well. And so I like now I'm really interested in like how do we get all of our natural teachers involved in helping us build AI products? But that's a little bit of a side tangent. It's really like a lot of us start out with like, oh, I'm just going to give it everything it could possibly need. And the challenge with that is that LLMs get confused. I know we talk about like million token context windows or 200,000 token context windows. But like when you start to bump up against that limit or even long before you hit that limit, the LLM is not good at following everything in that context. And so one of the keys of this first piece is how do I give it the right context at the right time? And this is why like MCP servers and RAG and like all these tools that are helping us feed in the right context at the right time are becoming more pervasive. And this is also a big part of the agentic flows is how do you equip the agent to pull in the right information it needs either through tools or through MCP servers or even RAG as a tool. And if people aren't familiar with those terms, like four months ago, all of this was like word, like acronym soup in my brain. Really, it's just like An MCP server, a tool is like an agent can request information from another tool, right? And that tool could be local in your system or it could be through a third party, which is where an MCP server comes in. And then RAG is just like you could have a database of documents and it's It's retrieving, thinking about it like a search, it's just retrieving the right context for that query, for that input. And so I put all of those in that context when, like that context engineering step of like, where is the right information to help the LLM do this task really well. The next step you highlighted is orchestration and not many people talk about this. Even I'm a little unclear in terms of where does discovery come in there? Yeah. Okay. So actually we didn't really get into discovery on the prompt piece. And I think where discovery informs the prompt piece, and this will also inform orchestration, is I can't teach an LLM how to do a thing that's supposed to help a customer if I don't know what the customer needs. Right? So fundamentally, like I'll start with my, I'll use my interview coach as an example. How do I give a student feedback on their interviews if I don't know what mistakes students are making? Right? So like I have my teaching knowledge. I know what a good interview looks like and I can start there. But as soon as I hit real data and I start seeing real student interviews, I start to learn like, oh, they're making these mistakes, not these mistakes. So now my... coach needs to know about those types of mistakes and needs to be prepared to give feedback on those types of mistakes. So even just how I'm instructing the coach in that context engineering piece is really informed by the opportunities I'm identifying in my interviews or by looking at students work. When we get into orchestration, this is where I recommend people start with the simplest solution to start and then when they through their observability, which is the next step, when they start to identify errors, then that is going to maybe affect their orchestration. So I'll give an example of this. My interview coach started literally as one prompt. It was just one long document of here's how we grade an interview. And I have a rubric where I grade interviews on seven different dimensions. And what I found was that the coach was starting to get confused across the dimensions. So like it'd be grading one dimension, but it would be pulling in instructions from another dimension. And I was like, okay, this is, it's great that my prompt fits in the context window, but it's getting so big the LLM isn't keeping it organized and keeping it straight. And I think this is a general thing that I see people write about a lot is that LLMs perform better when we give them a simple task. So I was like, okay, I have seven dimensions. I'm going to I'm going to break each of those dimensions into their own LLM call. So now I have a workflow. So what started as one prompt now becomes a workflow where I take the same transcript and I send it to seven different LLM calls and then I have to orchestrate the response. So I'm collating the responses before I send it to the student. So that's a workflow is just a series of orchestrated LLM calls. So that's one example of orchestration. Another example of orchestration is I could see in the long run like I might move to an agent model where the agent is smart enough to look at the transcript, maybe knows a little bit about my student's history. Maybe I'm going to pull in like the student's history through something like RAG or a tool and it's going to look at both and make a decision about what's the dimension that's most important to give feedback on. And instead of giving feedback on all dimensions, it gives feedback on the dimension that's most important to the student. I would put that in the orchestration bucket. Like how am I orchestrating how the LLM is going to respond to this input? And so where does discovery come into play there? I think orchestration and context engineering are really closely tied. It's how do we understand what opportunities we're trying to address and how are we giving the LLM the context and small enough tasks that it can meet those opportunities clearly and adequately. Okay. And what I'm hearing here is a really important focus on error analysis then, where discovery... and error analysis, it seems like are almost synonymous in the case of building AI features. And as you set up this good observability and you're doing your evals for quality and you're maintaining the system, you're going back into the context engineering and the orchestration to actually change things. Is that a fair summary? Yeah. So this is, let's get into this third bucket of observability. So Most people may not realize this because I didn't realize this and I think this is a huge ethical concern. Most AI products are logging your traces. So what does that mean? It means when you interact with the AI feature, they are creating a record of the inputs and outputs. This is good for the developer. I actually think we need to be much more transparent about this. I think this is a part where our discovery best practices of being upfront about what we're storing in our data practices is very relevant here. I think we need to be informing people this and maybe not buried in the terms of service. But this is a critical step. We do have to observe what these non-deterministic tools are doing. We need a way to log at least a percentage of our traces, if not all of our traces, so we can start to look at them. And you mentioned error analysis. This is where we're going to look at our traces, some percentage of our traces, and we're going to literally have humans review them and tag them were there errors. And I have been doing this a ton for my interview coach, where I literally look at the transcript, I look at the coach's response, and I write notes about what would I, as the instructor, have done differently. And I do this, I typically, transcripts are long, so I typically work in batches of like, 50 and then once I've done a batch of 50 I'm looking across my annotations and I'm looking for what are my common errors and that's telling me that's my feedback loop into do I need to change my context engineering do I need to update my prompts or do I need to update my orchestration and I've actually had errors that have driven both of those types of changes and then eventually error analysis leads to evals so some of my errors I can fix by just making prompt changes and they go away. Some of my errors, maybe I change a prompt and it fixes a little bit, but then they come back. So for errors that are kind of persistent, I'm going to write an eval. And an eval is just code. It can be code. It can be a data set. It can be another LLM. It's a way to say, how do I know how often this error is appearing in my traces? So it's an automated way of saying, how do I know how often this error is showing up in my traces? I prefer to use code-based evals or LLM as judge evals. And so what that means is once I identify an error, if it persists, if it's not easy to make it go away, I write an eval to detect the error. And then what that allows me to do is now I can A-B test changes. So I can have a set of transcripts that I use to run experiments. I can detect an error. And then I can change the way either my context engineering or my orchestration. And then I can run that set of transcripts on the old way and the new way. And then the evals is grading is the new way better. That was a lot. So I'm happy to pause for questions if there are some questions. Yeah, well, because we have limited time, folks, we did an episode with Hamel Hussain and Shreya Shankar on evals. If you want to go a little bit deeper on that. I want to ask Teresa really badly because she's been writing about Claude Code to just give us the 30 second introduction to Claude Code. Why? What is it? Most PMs are really scared of using it. Can they use it? Yeah. Okay. So we're recording this on a Monday. I will tell you, I am seven days into using Claude Code. But let me tell you what I did in those seven days. So about two weeks ago, I noticed an error in my interview coach. The error was an orchestration error. So by splitting my prompts into seven individual prompts, what it meant is the prompt, like one prompt didn't have any context for the other dimensions. And what I was seeing was they were all using the same excerpts. And so one section would say, this is a great question in the context of my dimension. And then in another section, it'd be like, this is a problematic question in the context of my dimension. And it was leading to really confusing feedback to my customers. And so First, I had to write an eval to detect the error. How often was this error happening? And then I had to identify a solution and then I had to AB test the solution. Okay, before a week ago, I did all of my evals in a notebook, in a Jupyter notebook, and I kind of did my own like AB testing ad hoc. Okay, here's what I did starting last Monday with Cloud Code. So I installed Cloud Code. It's a terminal. You install it in your terminal. You talk to it just like it's Claude, but the difference is in your terminal, it can see all your files. So I also started using VS Code a week ago. It's been a big week. So I started using a proper IDE. So I'm using Claude in the context of my development environment, which is VS Code. And I basically said, I've identified a problem with my interview coach and it can see all my interview coach code. And the first thing I want to do is detect it. And so like help me design an eval for this. And it gave me, sadly, a really complex design, which I got stuck on and spent like the first three days trying to get work on that problem. And then I went to sleep and I woke up the next morning and I came up with a way simpler solution. And I was like, hey, Claude, why don't we do this simpler solution? And it was like, great idea. Thank you, sycophant LLM. And it literally wrote all the code for me. Now I know how to code and I know how to read code. And so my rule when I let Claude code for me is I review all the code and I make sure I understand all of it. And then it's actually doing what it does because Claude code does make mistakes. So like you have to be a babysitter, but it literally wrote all of the code for my evals. For that eval, it wrote all of the code for my proposed fix. It also wrote code. like a testing harness for the AB test. Like it just did everything. I don't think I wrote a line of code all week. I reviewed a lot of code and I was like the architect because Claude solutions were often very complex. But if I think, yeah, I don't think I wrote a single line of code, but I released a major update to my interview coach and a brand new eval. And it was the first time I did an eval that spanned LLM calls. So it was like a more complicated eval than I had done. And I didn't write a line of code. There you go, guys. She started with it a week ago and she's already making huge changes. I cannot emphasize enough if you are listening to this podcast to get over the barrier of, hey, it's in the terminal. Even if unlike Teresa, you're not great at understanding and reading code and just get started with this tool. I think it's a game changer. I think it's even better than cursor. Before you go, Teresa, I have to ask because there's so many creators who are going to be listening to this podcast who have been looking up to you. You were one of the inspirations for them to even become a product creator. How big is the business of Teresa Torres? Yeah, so I'm a company of one, kind of a company of two. I have a full-time admin in the Philippines. She's technically a contractor. So from a U.S. standpoint, I'm a company of one. But I have a number of contractors who help with my business. We, in a typical year, we have 2,000 to 3,000 students a year in our courses. I know like in the era of Maven, that might not sound very big, but we teach all of our courses are 20 to 50 students each. We try to keep them small. We want a high instructor to student ratio. And for me, it's not really about like, I mean, I know like people talk about like I've worked with more people in discovery than anybody else, blah, blah, blah. It's not about size. It really is about impact. Like I know it's hard to build discovery habits. And I really all of our programs, we've designed them. to change behavior. And I think it's very different from some of these edutainment products that are out there. I know there's amazing classes on Maven. I'm not dissing Maven. I took the AI evals class on Maven with Hamel and Shreya and I absolutely loved it and I strongly recommend it. But we follow a little bit of a different process with our cohort courses. We practice, we focus a lot on really hands-on practice and a lot of support. So we're at, we've, we're at, we're at It looks like just over 17,000 students. Amazing. And how much does one of those courses cost? We have a few different formats. So we have our like product discovery fundamentals course, which covers all of the discovery habits, but think about it as an introductory course. When you cover all the habits, we can't go deep in any habit. That's a six week course. It has 12 hours of live instruction and it's $17.95 US. We then have a series of deep dive courses that are, they're each on a single habit. And so those are our skill-based classes. They're designed to build skill really quickly. So for example, we have one on continuous interviewing. This is where the interview coach shows up. And it's you round after round after round of practicing interviews. Our goal is for you to leave the class feeling really comfortable to do a real customer interview. Those courses are all $7.99. And then we are starting to convert some of our curriculum into on-demand courses. We have one today called Customer Recruiting for Continuous Discovery. We're about to launch our second one, Story-Based Customer Interviewing, probably by September. And those are $2.59. Okay, folks, you can do the math on that if you want to figure out what her business is. Teresa, as I said at the beginning, this was a true dream come true. And I think you over-delivered on the responses. Your fluency about AI is mind-blowing. Thank you so, so much for being on the podcast. Thanks for having me. This was a lot of fun. and i always believe in wishing in the future so i'm gonna go ahead and put it out there to everybody i hope we have a round to support this episode so teresa just has to come back thank you all for watching and see you in the next episode i really hope you guys enjoyed that episode it would mean a ton to me and the team if you could please subscribe on youtube follow on apple and spotify podcasts and leave a rating and review. Those ratings and reviews really help grow the show and help other people discover the show. And they help fund the production so that we can do bigger and better productions. Can't wait to share the next episode with you. Until then, see you later.

What happens to product discovery when AI can generate prototypes in minutes, synthesize interviews in seconds, and give you feedback before your coffee gets cold?

Does it make discovery obsolete… or more important than ever?

In this episode with Teresa Torres, legendary author of Continuous Discovery Habits, who has trained over 17,000 PMs across 100 countries.

She pulls back the curtain on:

- Why most customer interviews fail and how to fix them with story-based interviewing
- The real difference between testing your idea and testing your assumptions
- How to keep your Opportunity Solution Tree alive and evolving
- The five skills every PM needs to build AI features that actually work

If you’ve ever wanted to master continuous discovery and AI product development without drowning in fluff or hype… then this podcast is for you.

Transcript: https://www.news.aakashg.com/p/teresa-torres-podcast

Timestamps:

Teresa's Background - 0:00
Story-Based Interviewing - 3:20
Fake Discovery Signs - 4:08
Assumption Testing - 4:39
Continuous Discovery Framework - 5:35
AI Changes Discovery - 8:01
AI Synthesis Concerns - 9:21
AI Prototyping Era - 12:45
Ads - 15:45
AI Prototyping Workflow - 17:32
Common Interview Mistakes - 22:24
Interview Synthesis - 24:26
OST Updates - 28:53
Discovery Theater - 30:52
Ads - 32:15
Real Product Management - 34:03
AI Product Discovery - 35:29
Context Engineering - 39:16
Orchestration Explained - 42:03
Error Analysis - 46:01
Observability & Traces - 46:05
Claude Code Demo - 49:15
Business Numbers - 52:56

Thanks to our sponsors:

1. Miro: The innovation workspace is your team’s new canvas - https://miro.com/innovation-workspace/?irclickid=VCiVcr1RbxycTNSy1219xzQHUkpxGiT7VWmDzE0&utm_source=Test%20partner%20account%20miro&utm_medium=cpa&utm_campaign=&utm_affiliate_network=impact&utm_custom=Aakash&irgwc=1
2. Jira Product Discovery: Build the right thing - https://www.atlassian.com/software/jira/product-discovery
3. Parlance Labs: Practical consulting that improves your AI - https://parlance-labs.com
4. Product Faculty's #1 AI PM Certification with OpenAI's Product Lead (get $500 off) - https://maven.com/product-faculty/ai-product-management-certification?promoCode=AAKASH25

Takeaways:

1. If nothing in your backlog changes and you never kill ideas, you're doing fake discovery. Real discovery should constantly reshape your product direction.

2. Stop asking "would you use this?" Instead ask "tell me about the last time you solved this problem" to get reliable, actionable insights.

3. When delivery becomes free through AI, discovery becomes MORE important to avoid overwhelming customers with incoherent features.

4. Break your ideas into underlying assumptions and test those individually rather than building full prototypes first.

5. AI can handle 60-80% of interview synthesis, but you lose critical context and differentiated insights in that missing 20-40%.

6. Building AI products is like teaching humans - you need the right context at the right time, not everything at once.

7. AI product discovery is heavily focused on observing traces, identifying error patterns, and iterating on prompts and orchestration.

8. Weekly customer interviews should load your brain with user context, making you a better human LLM for product decisions.

9. Map customer stories to opportunity spaces and update your OST every 3-4 interviews to keep discovery actionable.

10. Teresa rewrote her entire AI interview coach evaluation system in one week using Claude Code without writing a single line herself.RetryClaude can make mistakes. Please double-check responses.

👨‍💻 Where to find Teressa:

LinkedIn: https://www.linkedin.com/in/teresatorres/
X (Twitter): https://x.com/ttorres
Website: https://www.producttalk.org/?srsltid=AfmBOopiWRDhn3IXM55mP320CUnE6THriNiviDHcZvk1ToAYXp6c3FDj
Courses & Mentorship: https://learn.producttalk.org/?
Book: Continuous Discovery Habits: https://www.amazon.com/Continuous-Discovery-Habits-Discover-Products/dp/1736633309

👨‍💻 Where to find Aakash:

Twitter: https://www.twitter.com/aakashg0  
LinkedIn: https://www.linkedin.com/in/aagupta/
Instagram: https://www.instagram.com/aakashg0/

#ai #productdiscovery

🧠 About Product Growth: 

The world's largest podcast focused solely on product + growth, with over 180K listeners. Hosted by Aakash Gupta, who spent 16 years in PM, rising to VP of product, this 2x/ week show covers product and growth topics in depth.

🔔 Subscribe and like the video to support our content! And turn on the bell for notifications.