Teresa Torres on continuous discovery in B2B, AI prototypes, and synthetic personas

50 to 80% of the time, we're wrong. We can't trust that feeling, right? Even if it feels like this is the obvious right thing to do. I would say like in an organization, big or small, ideas are like ice cream and discovery is like broccoli. It's an unfair fight. The vast majority of people are not going to pick broccoli over ice cream. But just like with health, we see like we get better outcomes if we eat broccoli than if we eat ice cream. I'll tell you the biggest mistake people make is they try to change their organization. It's true. If you're an individual contributor on a product team, that's the wrong level to try to enact change. The level where you should try to enact change is with you individually. And then as we start to adopt these habits, other people in our organization start to get curious about how we're working. And that's when we have the opportunity to influence. For a product team, in a b2b context to be set up to be successful a few things need to happen welcome to site and podcast today is a special one we're going to talk about my beloved topic discovery and i'm honored to greet one of the most influential voices in modern product management and person who is demystifying discovery process teresa torres for those who for some reason are not familiar with rila she's author of essential reading for product teams continuous discovery habits. She spent years in trenches as product leader and now coaches product teams. So if you ever struggle with building features that customers don't use or felt like you're guessing on what to be next, today's conversation is going to be incredibly valuable. So I'm very thrilled to have you today, Teresa. Welcome to the show. Thanks for having me. I'm excited to do this. Yeah. To set the stage, let's maybe define the glossary for the beginning. We'll talk about discovery today. And discovery helps us to find questions what to build and the risk of implementation. But there are different types of discoveries. Like working on design agency, we mostly focus on initial product discovery. And I've talked to my peers working on products. asking what discoveries they run. And it's usually also finding the problem, finding the pain points of different personas, then planning the implementation, scoping and working through implementation. And your continuous discovery is a bit different. So can you explain how it stands apart and what are the key elements that make it stand out? Yeah, so my continuous discovery framework was designed for outcome focused teams. So they're not starting with a solution, they're starting with they're not even starting with a customer problem. They're actually starting with an outcome. So an outcome could be a business metric they're trying to improve. It could be a big strategic objective the company is trying to reach. It's really just as a team, what's being asked of us? What does success look like? And then from there, we're looking at, so that's sort of business focused. And then from there, we're looking at what do our customers need? And customer needs are infinite. right? We can spend our whole lives trying to address customer needs. So we're going to filter those needs based on our outcome. So that's why we start with the outcome because it tells us which needs we should pay attention to. So a lot of it is starting with an outcome, discovering the needs, pain points and desires of our customers that can help us reach that outcome and then discovering the right solutions. Yeah. And what are the main criteria for this continuous discovery to work? So first one, as I understand that Product leadership should delegate the authority and ownership for a product team to actually impact the product. And what else? So the key concept with even that first one you threw out is this distinction between empowered teams and autonomous teams. So some people hear like, oh, I'm being tasked to deliver an outcome. I should be able to do it however I want. That's not quite true, right? We work in a business context. We work with other teams. We need alignment across our teams. We need alignment with our business strategy. And so it's not the case that we just give a team an outcome and they can do whatever they want as long as they reach the outcome. Our leaders need to set, they need to not just tell us what outcomes are important, but they also need to set the strategic context. So mission, vision, strategic objectives. Think about it as like the way that we work here, the types of things that we would do, the types of things we wouldn't do. Our revenue model is even part of that, how we sell our product. that gives the team guardrails. So it's your job is to drive this outcome within the guardrails of that broader strategic context. And what's hard about that is in a lot of organizations, that strategic context isn't very well defined. And then it means that leaders get frustrated because teams suggest things that they wouldn't consider doing. They violate one of those guardrails, but the guardrails are implicit instead of being explicit. And so it's not just that teams leaders have to delegate to teams. It's that they have to set clear guardrails so that teams are empowered to make decisions that will fit in the broader business context. And that's often a very common gap. Great insight. That's something that I like right now in my project. Yeah, it's very common. And a lot of it is common because our stakeholders have it in their heads. Like it's implicit, but they forget how much isn't being communicated to the teams. And then I think the other things that need to be in place is the teams need access to customers. So in a lot of organizations, sales or customer success often acts as gatekeepers to customers, whereas we want the people that are building the product to have direct access to customers. Ideally, we have good instrumentation of our products and we have good analytics. I know close to like half of product teams across the world have zero analytics. That doesn't mean they can't do discovery. They certainly can. But definitely, as you get more advanced in your discovery efforts, having good analytics can help. We definitely will dive deeper into getting access to customers because it's not only about users, but actually who make decisions. But while we are on this kind of initial level, I wanted to understand what are the main obstacles for teams to implement this framework? Because first, we can kind of look through the lens of startups or smaller teams that actually are in position to have the ownership around the product, but still they do not follow the continuous discovery in terms of finding the opportunities. What's the main kind of hurdle or obstacle to implement that? Is it cost, timing, skills, experience gaps? I think startups are actually the toughest environment to do this well, which is a little bit surprising. It is very possible to do it in startups, but I think what makes it uniquely hard in a startup is two things. One is most startup founders don't start with an outcome or a customer problem to solve. They start with a solution and they have a very strong vision. And especially if they're a first time founder, they don't have the experience to know that oftentimes our first idea is very flawed and it takes a lot of iteration to get from a flawed idea to a good idea. And so a lot of first time founders have to go through this learning curve of realizing that like. It's a little bit like the don't tell me my baby's ugly, right? Like there's a little bit of this discomfort they have to grow through before discovery is really going to be effective. And even if they hire people on their team or they hire a design agency, even if they're like doing all the right activities to understand is their product idea going to work, it doesn't mean they're receptive to that feedback. And in my experience, most founders have to get knocked down quite a few times before they're receptive to that. There's a reason for this, like to start a company requires that you'd be a little bit crazy, right? And that craziness is what leads to startup success, but it also is what can get in the way of us seeing the flaws in our products. And so I think a lot of second-time founders, definitely third-time founders, start to learn this. It's a little bit easier when you have an experienced founder. I think the second reason why it can be harder in a startup is because you're racing to get to revenue before you run out of money. It makes it even harder to say to a customer who may not be your ideal customer, No, we're not going to do that for you. And so what we see startups do is they get pulled in a lot of different directions unless they have a very experienced founder who's keeping them focused. Yeah. And for those teams that are part of large organization, we can assume that there is this delegation of authority and ownership, or there are those feature teams that kind of just work with customer requests. Do they have anything in power to change the status quo? Yeah, so both in startups and in mature companies, I would say across the board, an overwhelming majority of companies are very solution focused. The organization talks about ideas, they prioritize ideas, they put ideas on their roadmap. Literally everything in the organization is geared around solutions. And I think what's hard about that is a lot of them are guesses, right? They're not incorporating customer feedback. There's not a clear understanding of customer needs. Sometimes there's not even a clear understanding of business needs. And a lot of this is just due to human nature. In human nature, it's really easy to jump to solutions. We all do it. it actually takes some discipline and some rigor to force ourselves to say, wait, why are we considering this idea? What problem are we trying to solve? What value does that create for the customer? What value does that create for the business? So this is a little bit of a discipline. It's like telling people to eat their vegetables. Like we all know we should do it, right? But like we prefer ice cream. And so I would say like in an organization, big or small, ideas are like ice cream and discovery is like broccoli. And it's a... It's an unfair fight, right? The vast majority of people are not going to pick broccoli over ice cream. But just like with health, we see we get better outcomes if we eat broccoli than if we eat ice cream. Yeah, I don't have a lot of words of knowledge. Yeah. And the good news is I think we're going through a time period where a lot more organizations are starting to recognize that we can't always just eat ice cream. We need to eat some broccoli too. Some teams take it too far. They say we can only eat broccoli. And that's not a great life either. You probably want to mix in some ice cream. So it's okay every once in a while to just build a stakeholder idea. That's not going to break the world. But your question of if I'm working on a product team, either in a startup or a big organization, and my company doesn't work this way, what should I do? I'll tell you the biggest mistake people make is they try to change their organization. If you're an individual contributor on a product team, that's the wrong level to try to enact change. The level where you should try to enact change is with you individually. So what are the things that you're doing day to day that could get you a little bit closer to a continuous discovery mindset? And this is uncomfortable for people because it's really easy to think like, I can't do this because those other people need to change. It's like we're giving up our agency. We're giving up our sense of responsibility. But if we flip it and we say. Well, what can I do? It might not be easy. In fact, if your organization doesn't work this way at all, it's not going to be easy. But if you flip it and say, well, what's like the first teeny tiny step that I can take? A mantra I use a lot is something is better than nothing. It actually is empowering. You're taking back your agency and you're saying, I do have some control over how I work. And all of us have some control over how we work. Like, I don't know a single human whose boss is standing over them all eight hours a day dictating what they do, right? So we all have some control. And the reason why that works is that, like, first of all, we do, we have to work within our sphere of control. We do control a lot of our day, especially as knowledge workers. And then as we start to adopt these habits, other people in our organization start to get curious about how we're working. And that's when we have the opportunity to influence. Whereas if we don't change our own behavior first and we just tell people you're doing it wrong, all we do is we get people on the defensive. They dig their heels and it turns into this ideological war and literally nobody wins an ideological war. I really like the idea, like start with yourself, like lead by example. I'm also curious on the human side. Is discovery something that you can actually learn? There's a lot of habits and skills related to discovery. So in my book, it's called Continuous Discovery Habits, and I outlined, I think, 13 different habits. Each of those habits requires a lot of different skills. So when you're defining outcomes, it's a very analytical, quantitative process. When you're interviewing customers, it's a very qualitative, empathetic, like EQ, like human to human process. When you're experience mapping, there's like a visual element as part of it and requires some drawing skills. When you're moving into identifying the assumptions your ideas depend upon, it's a very black hat or devil's advocate or disconfirming questions kind of exercise. So there's a wide variety of skills involved. I don't think there's very many people that are going to be equally good at all of those skills. And this is why we talk about discovery as a cross-functional team sport. right? Because your engineers are probably better asking disconfirming questions than anybody else on the team. That's a very common pattern and you want them helping to identify assumptions. Your designer probably, but not necessarily is the best drawer on your team and they're probably going to be drawing those experience maps for you. It's just one of those things where You want to look at like where are your strengths and where are the strengths of your teammates and how do you as a cross-functional team take advantage of each other's strengths and collaborate. So not just farm it out and let one person take the lead, but really use each other's strengths to come up with a better team perspective. Now I would like to shift more into a more practical sphere. The building block of continuous discovery is repetitive process of customer interviews to collect stories. It really resonates with me because my beloved colleague, she repeatedly reminded us that we are not users and we need to collect the context about like to build empathy. And I think it really resonates with the designers. But still, I think collecting stories is a bit different how people do interviews right now. And you mentioned in some either product talks or in the book that our research is right now tiered. toward validating rather than exploring. So can you kind of explain what's the difference and how this type of interviews is different from what currently teams are doing? Yeah, so I would say there's two key building blocks. One is interviewing customers and collecting stories, but the second is assumption testing. And I actually think these are both equally important. What we get from customer interviews is we're uncovering unmet customer needs, pain points and desires. We're learning about our customers. We're not exploring our solutions. We're not getting feedback on our solutions. We're really focused on how do I learn about who I'm designing for, who I'm designing with ideally? Where does our product fit in their daily lives? It's generative research. It's helping us see and learn and understand goals, context, needs, etc. Assumption testing is helping us to evaluate our solutions. So once we understand the problem we're trying to solve and we're starting to design solutions, we're running assumption tests to understand will the solution actually address this need. You mentioned one of the things that I do teach is for interviewing in particular, I teach a story based format of interviewing. So what that means is instead of going in and asking your customers, what do you like to eat for lunch? We're going to ask them, tell me about the last time you had lunch. And it sounds like a very subtle difference, but the first question invokes a very speculative response. Your brain is not, and not you specifically, like all human brains are not very good at summarizing our behavior or reflecting on our behavior. A lot of this is like grounded in our biology. Like our brain's job is to keep us alive. It's to keep our lungs breathing and our heart beating. It's not to do all this mental work that we think it is for, right? And so when we're asked a question like that, our brain generates a fast answer. But that fast answer doesn't necessarily reflect reality. So for listeners that are familiar with the book Thinking Fast and Slow, that's Daniel Kahneman's book. Daniel Kahneman wrote about multiple decades of his work with Amos Tversky, where they basically invented the field of behavioral economics. And really what they uncovered was this whole suite of cognitive biases. So mental errors that our brains make. And in that book, he introduces this metaphor of system one versus system two. System one is our fast brain. Your brain is trying to spend as little energy on mental processes as possible so it can focus on keeping your biological systems working well. The problem with that is that it's error prone. It can be error prone. So if I ask you, what do you like to eat for lunch? There's a number of cognitive biases that are going to interfere with your response. It's going to be things like recency bias. So like... what did you have for lunch yesterday? Or your favorite lunch, right? But maybe you don't have it very often. Or it might be colored by, you recently had lunch with someone you really like. And so that stands out in your memory, right? So there's a number of cognitive bias, or it could be influenced by like, what you hope you would eat for lunch every day. Right? So like your healthy aspirational self as opposed to your real self. And so when we're interviewing our customers, we have to be aware of this psychology and we have to learn how to ask questions that invokes memory instead of these like fast system one responses. Now, most people on product teams are not experienced researchers. We don't have deep roots in qualitative research. And so what I've found is that teaching people to collect stories is a really simple tactic. It's still a skill. It takes practice. We do have to get good at this skill, but it's a simple practice that everybody can understand that helps to get much more reliable feedback. Now, a lot of people don't do it because I think people don't respect interviewing as a skill. And even if they like read my blog and read my book and say, oh, I got to collect a story, they'll make a mistake. They'll say, well, what do you typically eat for lunch? And they think they're collecting a story. But as soon as I add in what do you typically eat for lunch, we're now back in that speculative realm. So like this really is you have to keep the participant grounded in a single instance. So what did you eat for lunch today? What did you eat for lunch yesterday? Right. And this what this does is memory requires system to using Kahneman's metaphor. We have to stop and think and remember. And as the interviewer, there's a lot of techniques we can learn to help them remember. right and then we want to walk them through all the detail of the actual story and this is what's going to make sure we learn about people's real behavior and not like their aspirational behavior or what they speculate their behavior is like i used to teach a workshop where i would ask someone to volunteer and like there is one example i think i included this in the book i asked a woman tell me um like what is your criteria when buying a new pair of jeans and she didn't even hesitate she said number one my number one criteria is fit And then I said, okay, what else? And she said, well, price and style. I'm like, okay. And then I asked her, tell me about the last time you bought a pair of jeans. And she said, I bought them on Amazon. And the whole room chuckles, right? Because how do you evaluate fit her number one criteria on Amazon? And so I asked her that. I go, well, how did you know if they were going to fit? And she said, well, I bought the same brand I've had in the past. I said, okay, that makes sense. And then I said, have you ever bought a pair of jeans that were the exact same brand and they fit differently? And she said, yes. And I said, okay, so a fit. And any woman who's bought jeans has had that experience, right? If fit is her number one criteria, I was like, well, why did you buy them on Amazon? Well, it turns out they were on sale. So her primary criteria was getting a deal. And her secondary criteria was the convenience of online shopping. And her third criteria was fit. But that's not what I learned when I asked her what's her criteria for buying a pair of jeans. And this isn't unique to her. I've done this in literally every in-person workshop I've ever done. We see it almost every single day. Our brains are very error-prone when talking about our own behavior, but they're less error-prone when we talk about specific memories. People are really bad at answering direct questions. Yeah, they are. And I also, while on the topic about deep understanding of the context, since we were working with B2B a lot, I always had struggles with connecting and user feedback. or kind of interviews from users, connecting it to customers, their need, because they're decision makers, and how it translates to product outcome. Because if we collect stories from users, how does it then get mapped to the customer expectation on the value? Because they make the decision and they kind of switch and buy and so on. And we need to satisfy their expectation on the value. So I think this not only testing and the derisking value risk is the hardest one, and how to work it in the context of B2B. So for a product team in a B2B context to be set up to be successful, a few things need to happen. The first thing is their outcome sets the scope of their discovery. Like a lot of B2B is helping us do our work, right? So it's all about like, how do I help you do your work better? And that's a lot of our B2B teams are focusing on end user work in the product. We often also have buyer teams. We have product teams that are focused on the buying journey and on the buyer specifically. So if you're on a buyer team, your customer is the buyer. If you're on an end user team, your customer is the end user. And the reason why I say this comes down to your strategic context is your outcome on the end user team needs to be framed in a way where it's directly tied to a buyer outcome. We're not like in B2B, we're not just trying to satisfy end users because it's fun. We're trying to satisfy end users because it's going to drive a renewal. And so part of that strategic context is there has to be a connection between our product outcomes and our business outcomes. And so if, for example, you're on a B2B team and you're focused on optimizing some process that people do in their daily job, there is a belief in the organization that optimizing that process is going to drive retention. or optimizing that process is going to bring in new customers who are struggling with that. Right. So part of the strategic context is making sure that our outcomes, even our product outcomes, our end user focused outcomes are supporting the overall business. And it's built on assumptions like you need to test those assumptions like this user outcome will drive business outcome. Absolutely. Oftentimes, the way that we test those assumptions is we try to impact that product outcome. And if it doesn't in turn drive the business outcome, then we need to look at what's the next product outcome to try that might influence that. I will say there are some teams that span the boundaries, right? So there are some parts of a B2B product that influence the buyer and impact the end user. And those teams, we usually put our most senior teams on those problems because they actually have a very complex ecosystem and they need to be interviewing both buyers and end users. But if I had a B2B company that had like 40 product teams, I probably have three or four of those teams and everybody else is either end user only or buyer only. I was in a position when our product leader created AI powered prototype with like backend and stuff. He demoed it to the customer. He got positive feedback and to me it sounds like confirmation bias. He made an assumption, he validated it, and now it's impacted our roadmap. But I just struggle understanding if that's the right thing to do for product teams to kind of validate with AI prototypes those hard decisions if it actually will convert the client on the B2B context. So my first question would be, okay, one customer liked it. What about the rest of the customers? Right? Yeah. So that's the first problem. We don't build our products for one customer. We build our products for the market. And a big question product teams need to be asking, like the way that I frame product work, what's the smallest amount of product we can build to service as much of the market as we can? There's a reason for that, right? One, we obviously want to serve as much of the market as we possibly can, but we don't want a big sprawling product because it's expensive to maintain. it leads to a terrible user interface, right? It's hard for our customers to learn. So really it's balancing smallest thing we can build to serve the most people. So already what I heard is we're reacting to one customer conversation. The second question I would have is like, okay, we showed this customer a solution. Did they try it out? Did they use it or were they just shown a demo? So we know from books like the mom test and literally everything in the universe of like getting feedback from customers. If we just show our shiny object, it's really easy for somebody to say, Oh, that looks great. And that could be they're just being nice. It could be it really does look great. But until they try it and get their hands dirty, they don't see the warts. It could be like it really does look amazing. Just like I really am going to go to the gym four times next week, right up until I start to get busy, right? It could be that it's tapping into that aspirational self and they wish they would use something like that, but it doesn't mean they would. So there's a lot of challenges with this. And this is what AI prototyping enabled. What you just experienced has always happened. The way it used to happen is your boss would talk to that customer or whoever it was in the organization would talk to that customer and they just riff on ideas and they would talk about that idea and it would come to you as a vague idea that you then had to turn into a product. What AI prototyping does is now anybody in our organization can build something that looks like a product. The good news is the more a customer sees something that looks like a product, the more likely they're going to see things that don't work for them. So odds are you're getting a little better feedback than you would have if they just stayed in vague words. But I still don't think it's the right way to go. Now, here's what I'm going to tell you. There will be times where it makes sense to start with AI prototyping. Sometimes we're going to build stakeholder ideas to build relationships. Like sometimes there are ideas that are obvious enough or simple enough or fast enough or earn a big account that it does make sense to just move forward with the solution. The challenge is it always feels like that's the case. And we know from data, like 50 to 80% of the time we're wrong, right? So depending on whose numbers you believe, some people say 80% of their ideas don't work. Some people say 50%. I personally believe it's closer to 80%. A-B testing tools tell us it's a lot closer to 80 to 90%. So we can't trust that feeling, right? Even if it feels like this is the obvious right thing to do. So that's where I'm starting to look at like, okay, well, how many customers are going to be impacted? How much work is involved? What's the risk? Am I willing to just gamble and build it anyway and take on that risk? Or do I feel like I need to mitigate some of that risk and do a little bit more discovery? I will share, I love AI prototyping tools. In fact, I just built a little utility on Lovable last night in 20 minutes. And no matter how much I use these tools, it still feels like magic. What I like them for is when we're assumption testing. One of the things that we teach in our courses is you want to prototype to simulate a specific moment. So we're starting with a very specific assumption. That assumption lives in a specific moment in a story map. our prototype is just of that moment. Well, we now can create that prototype in like three seconds. Vibe coding is amazing until it's not, right? For anybody who's played with these tools, like they're really powerful. You can create real stuff until the real stuff gets a little too complex and it breaks. But if you're using AI prototyping to create a very simple interactive mockup or prototype of a single moment to test an assumption. They're great for that. And so what used to take design work now takes like a sentence. So I love AI prototyping for two purposes. One, they're really great at creating little personal utilities. So if like there's something in your workflow that you just want to automate, they're really great at that. Like this idea of software for one, they're amazing at. And then I think in the discovery process, they're really great. for assumption testing? Well, there are two paths I would love to like first to dive deeper into a situation when stakeholders made decision and just like tell us what to do next. And another one, AI. So maybe let's dive into AI because you mentioned like using AI prototyping to drive assumption testing. And there are other ways how AI impacts the discovery phase. Like there is a post that I just saw today about trends in AI. and how it impacts the discovery phase. And I will quote them and maybe you can react to that. So first is learning by delivery over discovery. Paul Alharan's post. Paul Alharan, yeah. So maybe you saw this post as well. PM doing AI evolves and product trio blends. So can you dive deeper into those changes, how they impact the discovery? using AI tools? Yeah, I wouldn't frame it as discovery happens in delivery. So first of all, discovery always happens in delivery. It's not where I would start. Even if you can just press a button and your thing is delivered, it's not where I would start. There's a lot of reasons for this. Already with continuous deployment, we're overwhelming our customers. Things are changing all the time. They're not always changing for the better. Even if delivery is free, it's a very expensive way to learn. You're rewriting. product documents, you're getting your support team up and running on the new change. Plus, it's expensive for your customers to learn a new interface or learn a new feature. We don't want to incur all of that expense before we've done any discovery at all. So I do not, no matter how, even if it's zero dollars and zero time to ship a feature, I don't think we should learn in delivery. I mean, we do learn in delivery, but I don't think we should start by learning in delivery. So I disagree with that. His second one, can you remind me what the second one was? Metrics emerge from data. Yeah, so Pavo and I both took the same class on AI evals. The class was called AI evals for engineers and technical product managers. It's taught by Hamel Hussain and Shreya Shanker. It's phenomenal. I strongly recommend it. And this idea of metrics emerge from data comes from that. They teach for AI products to look at your traces and to do error analysis. And then that helps you identify metrics that you... write evals around. This is a really big topic and I know a lot of people are not familiar with this space. I will say to some degree our metrics have always emerged from data, but it depends on what metrics we're talking about. If we're talking about our business outcomes, they need to be derived from how our company makes money. That is what our leaders care about. They should be derived from your revenue model, not from user behavioral analytics. So I saw that and I didn't, I mean, I kind of get it because I took this class with him and there is one context where that is true, but I didn't like it as a broad stroke. One thing I wish Powell did a better job on, and he has actually gotten much better at this, so I will give him credit for that, is I wish that he attributed his sources. So the product trio blending, there was a speaker in our class. His name is Aman. I'm going to look up his last name really quickly. I can't really. Maybe I can give it to you for the show notes. He gave a talk where he talked about product trios and PM, like what do PMs do in the trio? What do engineers do? What do designers do? And how AI is forcing that to blend even more. And a lot of this is because evals is a part of an LLM app development process that requires both the product manager and the engineer. And in the early days, which is where we are right now, a lot of PMs are learning the engineering work and a lot of engineers are learning the PM work because sometimes it's easier to go faster for one person to just do the whole process. I did this myself. I took the class, I learned the engineering work to the evals, and I am now doing both the product work and the engineering work on my own AI product. I have for two decades now argued that the roles in the trio should blur way more than they do. I don't think this is a new trend at all. Like I think the more your roles blur in a trio, the better, the better you're going to work together. So I don't think that's a change. I agree with the sentiment. I don't think that's a change. And I would love it if you would include a link to Amon's work in your show notes. And then the one about PM should be doing evals. I 100% agree with that. And I'm actually going to be doing a lot more in this space to help. educate product teams on the role of evals and how to do them well and how to do them cross-functionally. And also it resonates with the access to customers and meetings. So sometimes you cannot sit on the table with like high net worth people who make decisions. And sometimes if they record this transcript, you can get it or like video recording, it'll be great. But sometimes it's just like a transcript and it's implied that you can understand the context. So that's one thing where I think that Although you can use AI to make the transcript or analyze it, it's not great. And even sitting and doing it in old ways, just read through and debrief, that's way better than just throw it into AI and what the team was working on or what people were discussing. So that's one thing. Another is using synthetic data. I've heard stories that sometimes teams can create kind of proto-persona using AI and even run some tech. tests with this synthetic persona or create synthetic data to not eliminate but like filling the gaps if they don't have enough understanding of the context or like business understanding so you can go and run your research and like filling the gaps that you miss so is it the like can ai help this to fill out those those gaps or what are not to do's for teams in using AI for discovery. So you mentioned sort of transcripts. Synthetic data and kind of testing with synthetic personas. Yeah. So actually a real, not a startup, maybe even a startup. I don't remember the context, but I've heard the story about that. Yeah. Okay. So let's tackle the three that I want to tackle is transcripts, synthetic personas and synthetic data, because I actually think those last two are two different things. So with transcripts, here's what I'll say. I've said this. I wrote a blog post not too long after ChatGPT came out that said something like, don't use generative AI to replace discovery with real humans. I still believe that. Now, the key is to replace discovery with real humans. That doesn't mean we can't use AI to add to what we're doing with discovery with real humans. So my point of view is this, this will probably never change. If you are building a product used by humans, you probably need to be talking to those humans and to be testing your solutions with those humans. Full stop. I don't care how good AI gets, that is probably never going to change. That doesn't mean we can't use AI in addition to that. I'm going to give a real common example that I think most people are going to be familiar with. A lot of product teams have access to Gong sales videos and sales transcripts and sales trends. That's great. Am I going to tell you you shouldn't use Gong data? No, of course you should use Gong data. For people that aren't familiar with Gong, it just records all of your sales conversations. It lets everybody on the team access them. It does a little bit of AI synthesis on top. It's a great product. Does that replace my own interviews with customers? Absolutely not, right? So what I like about Gong is I can now do my weekly customer interviews and build my own mental model of how my customers go about their lives. And on top of that, I can get this goldmine of data from my sales team. I can see where they align. I can see where they misalign. I can see where my sales team might need some feedback on overcoming objectives so I can help with sales enablement. I can see where I might be missing something in my interviews because customers, prospects keep bringing up the same stuff in sales conversations. And I might need to broaden my scope in my interviews, right? So it's additive. It's not replacing. It's not a substitute. I think for most of these things, I feel the same way. When Synthetic Users first came out, so Synthetic Users is a startup where they build these personas based on your target customer market, and they pitch it as like on their first homepage, it was like, recruiting is hard, talking to customers is hard, getting permission is hard, just talk to our Synthetic Users instead. That's billed as a replacement. I do not like it as a replacement. Now, does that mean there's no use of Synthetic Users ever for any purpose? I'm not sure. In fact, John Whalen is a researcher who's like dove deep on the use of AI and research, and he's actually a big fan of synthetic users. Does he use it to replace talking to humans? No, he uses it as an additive, right? So like if you look at the amount of work you do in your product day, you don't talk to users about everything. That's impossible. So maybe for the things that like don't bubble up to be important enough to your interviews, you get some feedback from synthetic users. Do you treat it as gold absolute truth? No, you treat it as input from an AI, but it's probably better than nothing. Right. But the key is don't let it replace your discovery with real humans. And then the third piece, synthetic data. So this synthetic data thing comes from LLM apps. So when we're building LLM apps, we need to be able to test that LLM apps AI features are non-deterministic. So most features we write code. There's n dimensions of things it could do. We can write unit tests for all those n dimensions, and we know the code does exactly what we wanted it to do. When we're building an LLM app, it's non-deterministic. There's an infinite number of things that it could do. It doesn't always do the same thing every time, and so we need a way to test it. And so the way that we test it is with evals. Evals are a way of how we're evaluating our LLM product. The problem is when we're new, when we have a new feature, we don't have a wide variety of production data set of how customers interact with it. And so we don't know how to evaluate how good our app is. And that's where people are using synthetic data to generate a range of post potential customer inputs to then test the app before they put it into production. I think that is a perfectly valid use of synthetic data, but it raises questions about how do we generate that synthetic data and how do we generate it in a way to make sure it matches what we might see in production. or that it matches our best guess at what we might see in production. And that's where I think a lot of our discovery can help. If we have a really rich understanding of the opportunity space, we can use our opportunity space to help structure the way that we generate synthetic data. Now, if we've never talked to a customer and we're just guessing what customers might ask and we're using guesses to generate synthetic data, it's probably garbage. Right? Yeah. We also kind of skipped the topic about Opportunity Solution Tree, and this is something that I really would like to tap a little bit at least. So Opportunity Solution Tree is a way to map business outcomes or like not business, but just product outcomes to opportunities, to assumptions. And we also mentioned that sometimes it could be situations in which I don't know, like there are memes when product managers come up to their team and say, hey, I was just on a stakeholder meeting or leadership meeting, and now we need to pivot or kind of create something that we haven't expected. So sometimes, as you mentioned, it's okay to follow the whim of stakeholders. But if this is a constant change or pivot and they don't kind of follow the... outcomes or opportunities. The team is working hard. They can invest time in discovering opportunities. But there are some external factors like potential customer or even this new technology wave with AI. And I'm sure that teams found themselves in the situation when they were kind of not demanded, but stakeholders told that we need AI. probably wasn't on the roadmap and it wasn't on the opportunity space at that moment. So it doesn't mean that they need to create new opportunity tree or fit it into the existing tree. So how does it practically work? Yeah, let's talk about this in the context of AI, because I think you're right. Everybody's going through this. So first of all, let's talk about an opportunity solution tree. It's just a visual to help your team stay aligned as you try to reach an outcome. So if you're familiar with decision trees. if there's your outcome at the top of the tree, it branches when you're mapping out the opportunity space. So opportunities are unmet customer needs, pain points or desires. So as you interview customers, you're collecting needs, pain points, you're mapping them out, you're giving structure to the opportunity space, and then you're choosing a target opportunity to work on. And as you choose that target opportunity, you're exploring multiple solutions and you're evaluating them with assumption tests. And that's just as you work your way down the decision tree, there's those different components. Okay, so ChatGPD comes out in what, November 2022? Took about a year. Companies started realizing this is a real thing. And suddenly, everybody's saying, what AI features are we going to build? Most companies, because again, we talked about earlier, humans, shiny objects syndrome, we're all talking about ideas, we're all living in the solution space. Executives probably came to product teams and said, we're going to build A, B, and C. Now, imagine that you were on a team being told to build A. Doesn't matter what A is, you're just being told to build A. and you had been doing continuous discovery. Does your stakeholder care about A or do they care about having an AI feature? They probably care more about having an AI feature. So if you're being asked to deliver A, but you already have a good understanding in the opportunity space, you can ask a better question. You can look at your opportunity space and say, which of these opportunities is now better solved by AI? And now you can start to negotiate with your stakeholders. You can say, look, I can build A. I'm not really sure how to fit that in my opportunity space. It's not clear to me what need it solves. Maybe you can help me fill in the gaps. That's a part of the conversation. Another part of the conversation is here's two or three other needs where I could see AI having a big impact. What if I explore those instead or in addition to? Now we're having a much better conversation, right? But if I have no understanding of the customer, no understanding of customer needs, and I just have a different idea for I want to build feature B instead of feature A. Now we're just in an ideological war. It's an opinion battle about is A better than B. But if I have a rich understanding of my customer in the opportunity space, it's not an opinion battle anymore. I'm bringing new information to the conversation. I'm showing that I know my opportunity space. I know my customer. I understand my outcome. I understand what matters to the business. And I'm in a much better position to negotiate and influence. Yeah, if we understand what pain points our customers have, then we understand how AI can solve that. I really love the story from Notion. They wanted to launch AI Assistant first, but after discoveries and iterations, they just created a button that magically sorted out the inbox. So that was for their mail client. So yeah, that was kind of an interesting thing. Eventually you come to that decision, but if you understand the clients and their needs, pain points and the opportunity space in advance, then you can come up with better solutions. I'm being mindful of the time, so maybe we can move to closing. And we usually have a section with rapid questions. It's not necessarily yes or no, but just like what's on the top of your mind. What's the most surprising thing you've learned from coaching teams? Something you haven't expected? That's a really good question. I think one thing that surprised me is at large companies, how little work is happening. Teams spend all day every day in meetings. I'll give a really specific example. I know we're supposed to be quick. I had a team that their customers were clinicians, doctors and nurses. They were having a really hard time finding doctors and nurses to interview. I asked them, I said, doesn't your company have a clinician team? Can you just go interview a colleague? It took six weeks to get on a clinician's calendar, even though they were coworkers. And it's because they're literally booked in meetings. Some of them are double and triple booked in meetings all day long. I know that meeting culture is terrible and that this is a problem for everybody. I had no idea it was that bad. What's the red flag that tells you the team isn't ready for continuous discovery? They're still stuck in territorial battles, right? I do this, you do that. We don't talk to each other. We just hand things over the wall. Yeah, cross-functional team is the best. One tool that every PM should use. Product manager, I mean. CatGPT or Claude. Easy. And best question to ask a customer when you're completely stuck. Story-based question always. Tell me about a time when. Yeah. And as we wrap up, I would like to hear your thoughts about the trends, like about software development lifecycle in general, because everything is collapsing. And what are your predictions? Will there be a time when AI will do discovery as well? I mean, we need to talk about customers. people, but in terms of pinpointing potential opportunities or flagging something into your face when there is, I don't know, some trends in data automatically. So what are your predictions about Impact API in general? I agree with what Marty wrote in his essay about this. I think the easier it becomes to build things, the more important discovery is going to become. And I think there's a really great analogy for this, for the people that have seen the Simpsons episode where Homer Simpson designs his own car. It's like a Frankenstein vehicle with amazing features and no coherence. It's just silly. And I think that's what we're going to run the risk of is when anybody in the organization can decide to add something to the product and it's effortless, we're going to end up with really unusable products. And I think we're already seeing that. We're already seeing a ton of companies tack on AI features without thinking about customer problems, without thinking about quality of those AI features. In fact, I posted over the weekend asking on LinkedIn, what's your favorite AI feature that was added to a non-AI product? And very few people have replied because I think very few people have a good example of this. I bet if I asked the opposite, what's your least favorite, I would get hundreds of replies. because we're in this like day one garbage, like everybody's seeing it as an arms race and people aren't thinking enough about quality. So I actually I just think we're I don't think the basics are changing. In fact, I think they might be becoming more important. Yeah. And also like speed of delivery is so fast, like you can have like multiple changes and you're just already frustrated. Like I remember that this button used to be here and it just will amplify. All right, I would love to end this conversation with your quote, which I love very much about most of work in discovery is not following the process, it's managing the cycles. So it just like have a continuous improvement mindset, start with something small, iterate, improve and just like it's not there is no silver bullet. or discovery. Yeah, I think the key is to not think about the book as a recipe, but just think about it as guardrails. It's habits. I mean, I put habits in the title for a reason, right? So it's just how do we build good habits so that we're always learning. It's all about fast feedback loops with customers. Yeah, Teresa, it was very incredibly valuable. Thank you so much for sharing so many insights. And it was a great pleasure hosting you today. Thanks for having me. It was fun. And to everyone who's listening, if you found this episode helpful, please subscribe and leave a comment what resonated with you the most. I would love to hear about your discovery processes and how you develop your discovery skills. Woohoo! Yay!

In this episode, we chat with Teresa Torres, one of the most influential thinkers in modern product management, about embedding continuous discovery into your product practice.

Teresa is the author of the essential book Continuous Discovery Habits and a world-renowned product discovery coach. For leaders struggling to bridge the gap between company goals and customer needs, this episode offers a masterclass in building products that matter.

Teresa unpacks the nuances of discovery in complex B2B environments, the cognitive biases that derail customer interviews, and how to leverage AI without falling for the hype. This is a tactical guide for leaders who want to move beyond feature factories and build outcome-driven teams.

Highlights:
00:00 Coming up
01:04 Introduction
01:53 Understanding continuous discovery
02:34 Key criteria for continuous discovery to work
06:09 Why startups struggle with continuous discovery?
08:20 Reasons mature companies fail at discovery 
10:33 #1 mistake in discovery
12:12 Learning path to the effective discovery
14:31 How to interview users to get their true needs
17:36 Teaching teams to collect stories
20:23 The buyer and end user problem in discovery in B2B
23:28 Real use case of AI prototyping
28:56 AI's impact on discovery: Paweł Huryn's post
33:08 AI transcripts, synthetic personas, and synthetic data in discovery
39:39 Opportunity solution trees and AI
44:11 Rapid round of fun questions
46:20 The future of AI in product development

Mentioned in this episode:
– Aman Khan on the Blending of Roles in a Product Trio: https://www.youtube.com/watch?v=XueTa4qrMpg
– AI Evals for Engineers and Technical Product Managers Course: https://maven.com/parlance-labs/evals

If you find this content valuable, please share it with your colleagues and friends. It will help us broaden the influence of Teresa’s messages to those who need them. Thank you!

Let’s connect on LinkedIn:
🎙️ Ana Mudryk⁠, podcast host, Head of PdM at Cieden - https://www.linkedin.com/in/anastasiya-mudryk/edit/forms/next-action/after-connect-add-position/
🎙️ Teresa Torres, product discovery coach, author of Continuous Discovery Habits - https://www.linkedin.com/in/teresatorres/

This podcast is brought to you by Cieden, a digital product design agency. If you are looking for product designers, let’s chat: https://bit.ly/3XNY56r