Why There's No Single Right Way to do Discovery - Part 3

Hi everyone, welcome to why there's no single right way to do product discovery. I'm going to give everybody a second to get settled here as I see people join in. I will go ahead and start with introductions while everybody's getting settled. So I'm Teresa Torres from Product Talk. I'm joined today by Hope Gurion. Hope, do you want to say hello and give a little introduction? Yep. Hi everybody, thanks for joining. I know people are still... Getting signed in and hopefully getting comfortable for our discussion today. I am a former chief product officer and now coach and advise products leaders and teams and a partner with Teresa for a number of years and really I've found a lot of value in these methods so excited for all of your questions and to share more information with all of you today to get even better at your discovery practice. Perfect. Thanks Hope. And then for those of you that don't know me, I've been working as a product discovery coach. for the last six years, teaching cross-functional product teams how to do continuous interviewing, to discover opportunities, and run rapid experiments and rapid prototyping to evaluate solutions. And I blog at ProductTalk.org. Hope and I are joined today by Melissa Suzuno, who is my blog editor, who also helps out with a variety of content marketing activities for ProductTalk. Melissa, do you want to go ahead and say hello? Sure. Hi, everyone. I'm excited to be here today and look forward to hearing all of your great questions that come in throughout the webinar. Yeah, great segue, Melissa, because let's talk a little bit about how the webinar is going to work. If you're new to Zoom webinar, we have a few resources to point out. First of all, you have a Q&A window. So at the bottom of your screen, you should see a button for Q&A. If you have a question over the course of the webinar, we will be taking questions at the end of each topic and at the very end of the webinar. So if you want to submit a question for Hope and I to address during the hour, please go ahead and submit it in the Q&A box. We know from past webinars that we get far more questions than we can answer during the hour, but I will share that we are going to tackle some of the questions that we aren't able to get to afterwards via video, and we will be releasing This is a three-part series. This is the third part. We will be releasing all three parts plus additional Q&A via the Product Talk YouTube channel. So look for that in the coming weeks. And then there is also a chat box. So to help make sure everybody can find the chat box, if you look down at the bottom of your screen, you should see a button for chat. If you're not seeing it, it might be buried under that More menu. But go ahead and in the... chat box tell me where you're dialing in from just so we can get a sense for who's in the room and where you're what part what parts of the world are represented all right so we have atlanta georgia toronto new york germany it's going too fast for me to read croatia all over someone from portland oregon which is where i am currently at south paul brazil paris boston boston hopefully you're staying safe I know you're having a horrible outbreak. Auckland, New Zealand, Lake Forest, Toronto. Excellent. Toronto, where I'm from. If you typed London into the Q&A chat, I'm not going to name you. This was a test of can you find chat versus Q&A. Chat is where we want you to connect with each other as we give the webinar. If you... want to just tell us what resonates with you, that'll help us feel like we're not just talking to ourselves, put that in chat. The Q&A box is going to be for the questions you want us to address throughout the webinar. So if you have a question you want us to address, it goes in Q&A. If you want to just give us a shout out, say something resonates, connect with your peers, put that in chat. And we'll help steer you in the right direction if you get it wrong. So no worries, we're not going to... Make any judgments or grade you based on which window you put things in. All right, so I think that is all we have from a housekeeping standpoint. So I did mention this was a three-part series. We've been talking about why there's no single right way to do product discovery. Hope and I both coach product teams. We hear, especially from the teams that are really eager to learn, we hear a lot about, am I doing it right? And we also hear, is this method better than this method? We see a lot of dogma in the industry around the one true way to do things. And I think what we've learned from working with teams is that there's actually a lot of ways to do this. And what's most important is that we find the right fit for the right team so that it's sustainable over time. And so Hope and I put our heads together. We started thinking about, as a product leader, if you're trying to help your teams become, adopt more continuous discovery practices. and you want to give them the flexibility to work the best way that works for them, what are the key underlying principles that you should be looking for to evaluate if they're doing discovery well? And so that's what we've been discussing in part one and part two. In each webinar, we've tackled a couple of principles. I'm going to do a quick review of what we covered in like three minute quick review of what we covered in part one and part two. And then we're going to dive right into part three. As I go through the part one and part two principles, if you're like, wow, I missed that and I really wanted to catch it, again, they will be shared on the Product Talk YouTube channel. All right, so in part one, we covered collaborative decision making. This is the idea of the product trio, a product manager, a designer, and a tech lead working together and owning product decisions together. That's sort of the first principle we look for. The second one is to externalize your thinking. We talked about opportunity solution trees, customer journey maps, experience maps, story mapping. Lots of ways for teams to externalize their thinking so that they can align around it and so that stakeholders can follow progress. So that was the second principle. The third principle we talked through was being outcome focused. So this distinction between outcomes and outputs. How do we get teams to not just focus on shipping code but to look at the out... outcomes, the impact of that code, and the outcomes that are being driven. That was part one. Part two, we dove into making sure your teams are discovering opportunities. So we teach using customer interviews to discover opportunities. We know some people like jobs to be done as a way to do this. We know some people have the luxury of doing customer observations on a regular basis, but what are your teams doing to make sure that you're constantly exploring customer needs, pain points, and desires? And then the last thing we covered in part two was talking about prioritizing in the opportunity space rather than the solution space. So rather than taking a whole bunch of solutions and ranking them against each other, comparing apples to oranges, first making that strategic decision in the opportunity space and saying, which problems are most important for us to solve, which opportunities would have the biggest impact on our outcome. So again, if you missed part one or part two and you want to dive deeper into those principles, videos will be coming soon on that Product Talk YouTube channel. And we are going to go ahead and dive into the next two principles. So the way this is going to work, we're going to introduce a principle. Hope and I are going to talk about it for a few minutes. Feel free to submit questions about that principle. We'll take a couple of your questions. Then we're going to go on to the second principle. We'll do the same thing. And then any remaining time that we have left, we'll tackle any questions. If you made all three parts, you can ask about any of the principles. And we may prioritize questions on today's principles. But feel free to throw it all into the Q&A chat, and Melissa will sort it all out for us. Alright, so today's first principle, we are going to talk about compare and contrast decisions. So this is one that I think not a lot of people think about when they think about discovery. It comes from decision-making research. So we know from decades of research on what leads to better decisions that when we're trying to solve a problem or we're trying to meet a need to consider more than one option. And so what we see product teams do is they tend to get stuck on their favorite solution. They pursue one at a time. They work on one customer problem without considering what else is out there. And we get really boxed in to what is often referred to as a whether or not decision. Should we build this idea or not? Should we solve this problem or not? Whereas decision-making research tells us we're actually better off setting a compare and contrast a team. Hope, do you want to jump in and share a little bit about what you've seen with teams on this? Yeah, I think, you know, there's a spectrum, there's an evolution. And I think that sometimes when teams are first doing discovery, the fact that they're considering more than one way to solve the problem is a great start. But it can still sort of devolve into, like, how do we know whether we're not we've made a good choice? Or you... or I see teams running like endless experiments without clear success criteria for those experiments. And so I think that why this principle is so impactful for product teams is when you're explicit about how you're going to judge one versus another, you can have more productive conversations and you can be more focused in what you need to learn as quickly as possible. And I think that's why framing decisions as compare and contrast can really help accelerate teams from feeling like they're in an endless loop of research without knowing, like they're expecting some epiphany to just appear. Yeah, this is great. So actually, I think there's a lot of ways to frame, compare, and contrast decisions, and we can get into those. But I realize we have a poll for everybody to kind of get a sense for where you're at in your company. So let me go ahead and launch that. and give everybody a chance to sort of respond to this poll, just about how does your company frame decisions? So choose the option that best reflects what you do at your company. And then we'll try to tailor our responses based on what we're seeing from you so that we can dig into what matters most for those of you that are joining us. All right, so it's looking like... Seen a lot of responses in the first category. We work with one problem or solution at a time. A little bit around we debate until we all agree and then a few smattering from the last two. Okay, so They're all pretty close though. We're pretty evenly spread. Let's go ahead and share those results. Okay, so let's let's let's talk through this a little bit. So The visual that you're looking at here is a visual of a tree structure that I like to use with teams. It's called an opportunity solution tree. It helps teams map out their path from an outcome all the way through to what should we build. And so what I think is nice when we're talking about compare and contrast decisions is that it helps us think about where to compare and contrast. So if you're working on an outcome, the first thing to compare and contrast is, What are the, how do we compare and contrast the opportunity space? What opportunities are available to us? If we were to try to drive this outcome, what customer needs, pain points, desires should we be addressing? And so instead of tackling the first problem you hear about, or what we see in a lot of companies is this ping ponging back and forth based on the last customer you talked to. They showed a need, you jumped to solve it. You talk to the next customer, you jumped to solve their need. but instead taking an inventory of the opportunity space and comparing and contrasting and saying what's the most important thing that we can be doing. And then once we choose a target opportunity, we can also compare and contrast at the solution level. So Hope talked about this a little bit. When we're experimenting, if we're only working with one solution, it's hard to evaluate our experiment results good enough or not. Whereas if we're comparing and contrasting solutions, we can ask a different question, which is which of these looks most promising? Which looks best? Is there a clear front runner? I'm going to stop sharing that. Hope, you want to add anything there? I mean, I'm curious to see what people have questions around, but I will say when people are debating, so just looking at the poll results, right? Sometimes there's an element of debate. Sometimes there's an element of desire to get to consensus. And sometimes you're just working with one opportunity or solution at a time. And I think like there is, it will actually help you get to more decisive calls if you frame your criteria for choosing amongst these. And if you're going to debate or trying to achieve consensus, better to get it at that level, focusing on the criteria that helps you compare. one of these opportunities or one of these solutions versus another. And then you'll be better set up to revisit that criteria when you actually have data to make an informed choice. And so I think that's where there's just, there's often that step is just missing from the way product teams and frankly their stakeholders are deciding. It's not explicit for a lot of teams. Yeah, I think that's a really critical point here, which is Your outcome is helping you determine what's the best opportunity to go after. It's the one that's going to have the biggest impact on your outcome. So can we agree on what the outcome is and then agree on how we might assess those opportunities so that it's easier to come to a decision? And then same with when you're exploring solutions, have you chosen a target opportunity and your evaluation criteria is? how well do those solutions address that target opportunity? And of course we can answer that question with prototyping and experimenting. All right, why don't we turn to a couple of questions. Melissa, what do you have for us? Sure. One of the questions that's come up a couple of times is about the criteria that you consider. And so people are just wondering if you could define exactly what you mean by criteria or give an example of the types of criteria that you would consider. Sure. So I think the ultimate criteria is your desired outcome. So we talked a little bit in part one about each team kind of working towards an outcome. This could be if you're a consumer site and your primary customer metric is engagement, you might be looking at evaluating the opportunity space based on what's going to drive engagement the most. There's a lot of ways to measure engagement. So you might have teams focused on things like DAU, which is daily active users. Some even get more sophisticated and divide daily active users by monthly active users. You might be looking at engagement as month-over-month subscription retention. How you're defining that outcome becomes your ultimate criteria. But then as you work through the discovery process, it might get even more and more refined. You might be looking at how well does a solution address a given opportunity. And so that's often a stepping stone towards your outcome. So we've identified a need. We think if we solve the need, it will drive the outcome. So now our criteria is how well does the solution address the need? And so what that criteria would be would be really dependent on what that need was. But if the team is discovering the need together and has a shared understanding of that need, it gives you a jumping off point for a shared criteria. Hope, do you have examples? Yeah, I'll give a couple of examples. So these are, when you're at the solution level, let's say you're, a lot of people work in, you know, some sort of transactional, let's say e-commerce website, right? That you're usually, you might be looking for ways that you can drive more e-commerce revenue, changing the purchase path, changing your product mix. There might be many paths that you're considering, many opportunities. to increase revenue from your customers. Assuming you've picked the opportunity, let's say it is streamlining the purchase path. You feel like that, whatever you've learned in your discovery, it's too cumbersome, it's too complicated, it doesn't have the payment methods I need, whatever the things that you heard from your customers led you to believe that you need to now be investigating solutions for how to improve the checkout process. Typically what you define again if your outcome maybe is revenue you're looking for something that is showing that you've actually increased maybe purchases per visit or you could be looking at you know checkouts per add to cart. Again it really depends on how you define your success for your team. But when you're doing that what I have seen some teams do is they'll create you know maybe three or four different designs and they just run them in an A-B test looking for a winner. Now, on the one hand, they're making a compare and contrast decision because they're waiting for a winner. But without explicitly saying, this is the line in the sand that we're drawing, it must not only increase, say, checkout rate, but it must not lower average order value or something like that. If you aren't clear and explicit or it must if it's not going to move the needle by at least 5% or 10% and have you know no negative impact on our average order value you haven't set explicit criteria to figure out which of your potential designs has the best chance at meeting that criteria for success. And so once you've defined that explicitly you can it actually also will help you think critically about your solutions and whether or not they are even on the right track to deliver on that so that you can get qualitative feedback or quantitative feedback to figure out if, in fact, they've met your success criteria for that solution. Yeah, let's stick with this e-commerce example for a minute. So I know a lot of people in the poll. responded with, we work with one problem or solution at a time. So whether you're explicitly outcome focused or implicitly outcome focused, if you know, even if it's not explicit, but you know generally you're trying to increase shopping cart order size, right? And you know that that's really like what the company cares about right now. What a lot of teams do is they start generating ideas like and they just fill their backlog. Here's what we're going to do. So I think one of the easiest ways to go from to get to compare and contrast decisions, if you're just working with one solution at a time, is to get in the habit of asking what else could we do, right? So we have this great idea. It's the top of our backlog. We're about to start working on it. How do we bring in this idea of opportunity cost? Every time we do something, we have the opportunity cost of all the things that we can't do. So how do we start to service those so we can compare and contrast? So we have feature A that we love, that we think is a great idea. What else might we do? So you get feature B and feature C. And you can do that at the opportunity level too. If you talk to a customer and they tell you about a new problem or a pain point and you're really motivated to solve it, take a minute and just say, what else are we hearing from customers? How do we compare and contrast? Should this really be the thing we work on next? Okay, and next question is about how long to spend on each step. So they're asking, you know, how long do you typically spend debating opportunity? How long is on solution? When do you move on from prioritizing the opportunity to the solution? So if you could share a little bit more about, you know, how much time you spend on each step? Yeah, that's a hard question. So like many things in the product world, the answer is it depends. One of the things I encourage teams to do is to really think about their decision points as two-way door decisions and then to quickly follow them up with experiments to see if they got it right. So let's break that down a little bit. The idea of a two-way door decision came from a Jeff Bezos Amazon shareholder letter. He used level one and level two decisions, which is a little less clear. The concept is the same, one-way door, two-way doors. The idea is that if it's a one-way door decision, you walk through the door, you see the consequences of having made the decision, and if it turns out you made the wrong decision, the door is closed behind you, you can't come back. Whereas in a two-way door decision, you walk through the door, so you get the benefit of having made the decision, you see what's on the other side, if it turns out you don't like it, you can easily turn around and come back. A lot of the decisions that we're making in discovery, what problem should we solve? solutions might work. If we're doing fast iterative discovery cycles, they're two-way door decisions. So we don't want to spend a lot of time debating and deliberating. We want to be data informed and we want to make the best decision we can given what we know today. But we want to make a fast decision and then follow it up with verification through prototyping and experimenting. Okay, great. And you want to take one more question on this topic before we move on? Okay, so this question is about the opportunity solution tree and basically they're wondering if you track all your decisions in one mega tree. When talking with customers, you might get dozens of opportunities per interview, so keeping them organized in a tree seems like it would be difficult. You want to tackle that one, Hope? Yeah, I'll tell you what I recommend because you're right. If you're doing customer interviews continuously, you could end up with hearing many diverse points of view from customers. And you might say, well, okay, what is the cost of me trying to capture all of this in the tree after each interview? Or what I recommend teams do, especially if you're keeping these snapshots, which shouldn't take long to capture, so you capture the opportunities. What I tell teams to do is if they've got them easily accessible, and a lot of teams are using Miro or something similar today, you've got your snapshot. Decide as a team how often you want to commit to revisiting the opportunities on your tree. And especially if you're thinking about pivoting from the opportunity you're working on to another opportunity, that's a good time to consider refreshing your tree. But then you can just sort of look at, you know, are there things that we're hearing number of times that we think are absent from our tree. You can decide a rule of thumb. If we hear something five times and it's not on the tree, we want to try to make sure that that's captured in the tree. If you feel like you're hearing diverse points of view that are actually representing different segments of customer needs, you might want to branch by customer segment and make sure that's reflected in your tree. I don't think teams should be trying to have a perfectly maintained up to the minute tree. It is really meant for you to facilitate good decisions and alignment with your teams and your stakeholders to achieve your desired outcome. So it's really what's the right level of investment for you to keep it up to date so that you're making good decisions. Yeah, I want to clarify a couple things Hope said because she used some terminology you might not be familiar with. One of the things that we One of the things that we teach is to create interview snapshots. They're just a one pager that summarizes what you heard in a single interview. What's nice about that item is that you don't have to worry about patterns. You don't have to worry about, am I hearing this from other customers? You're just capturing as much as you heard in an interview. So the question asker said, what if I'm hearing dozens of opportunities in a single interview? All of those would go on the snapshot. We then encourage teams to map out the opportunity space. And when you're doing that, you're looking for patterns across your interviews. So you wouldn't include everything. You would look for the common traits. Now, Hope and I teach teams using interview snapshots and we use the opportunity solution tree. But I really want to emphasize there's not one way to do this. So if you don't use snapshots and you don't use the opportunity solution tree and you use something different, that's fine too. The key here is instantiating these principles. So one of our principles was to externalize your thinking. We do it via opportunity solution trees. You might do it through impact maps. Maybe there's another. Maybe you do it through a Kanban board. Whatever works for you. But the key that we want to encourage is that you're making sure you're externalizing in a way that allows you to compare and contrast. Are you considering more than one opportunity? Are you considering more than one solution? All right. Why don't we do this? So if you have questions in the queue that we have not gotten to, we will continue to answer compare and contrast decision questions at the end. But we are going to introduce our second principle for the day, which is around surfacing and testing assumptions. This is, we think, a big common in the vernacular in the industry, but far less common in practice. We want to dig into a little bit around why, but before we do that, I'm going to remember to launch our poll so we can get a sense for where you're at with how to use surface assumptions in your product practice. And again, choose that one that best reflects what you're doing today. All right, so it's looking like a lot of you discuss assumptions, which is great. A few of you don't do this yet. We'll help you get started with that today. And then a smattering are explicitly externalizing them or testing them. Let's go ahead and share those results. Okay. So let's dig into this a little bit. Maybe Hope, do you want to start with for those, for folks that aren't doing this today, do you want to talk a little bit about why this is so important? Yeah. I find that not enough teams do this. And it's so, it will... transform the way you think about your responsibility and the ability to learn quickly, but if you really make this a critical part of your discovery practice. It is often that when people are working in the opportunity space or working in the solution space, they have in their minds many many beliefs and assumptions and Some of these assumptions could be right and are totally reasonable. Some of these assumptions may be completely biased and based on a very small, unrepresentative sample of their prediction of future human behavior. And what we need to do is to take it out of our brains and put it into a place where we can examine them. And by doing this, we can see how well we understand one another and our perspectives on what we think. going to be critical to be true for our solutions to deliver on our expectations and deliver on the opportunity and we just it's not a common practice it's not something that people do it might feel awkward and strange but once you do it you can actually start to see and really break it down into if this is not true then it will really decrease the probability of our success in meeting our customers' needs and delivering on our desired outcome. And that's why it's so important for this to be part of the team's practice. Yeah, I think the other thing it does is it really unlocks fast iterations. So I see a lot of teams, they work at the idea level. So the feature level, the solution level, we want to build this thing. And they get stuck with how do we test it? And we see an over-reliance on A-B testing to test whether it's the right thing to build. And the problem with that is that you just did all of the work to build it before you learned if it was right or wrong. So A-B testing is great as a measurement tool, but it's not the best discovery tool. Unless we're looking at early in the stream 404, like smoke screen type A-B testing. It's great for marketing and landing pages, obviously. But really, we want... What gets us out of this trap of testing the whole idea, whether that's through A-B testing or taking a couple weeks to prototype the whole idea and then running a participant through the whole prototype, if we can take the time to surface the underlying assumptions, what needs to be true for this idea to work? And really, we tend to see assumptions fall into desirability categories. Why are we assuming people want this? There's usability assumptions. What are we assuming people are able to do? Will they understand it? Can they find it? There's viability assumptions. Is it good for our business to build this? Is the effort worth the reward? We also like to look at desirability, feasibility, what am I forgetting? Feasibility assumptions. Thank you. Can we build it? Yeah, so feasibility assumptions is... Can we build it? Is it technically feasible? But it's also, is it feasible from a compliance standpoint, from a security standpoint, from a company mission and vision standpoint? Like, is this something we, our specific company, can build? And then finally, we really encourage teams look at ethical assumptions. What are we assuming about potential harm? How are we using data? Do our customers understand how we're using data? Do they understand, would they be okay if they knew how we were doing it? And so using those five categories, so desirability, feasibility, usability, viability, and ethical assumptions, really helps teams to deconstruct an idea into its underlying assumptions. And then almost always, it is way faster to test a specific assumption than to test the whole idea. And so by testing assumptions, it allows us to test multiple assumptions in the same week. So when we hear things like Marty Kagan says, the best product teams run dozens of experiments a week. If our experiment size is an A-B test of a full-blown feature, it's impossible to do dozens of experiments a week. But if we're testing teeny tiny assumptions, and we can test them in a couple hours or a few a day, it's easy to get to dozens of experiments. Hope, do you, I'm going to put you on the spot a little bit, feel free to throw it back to me. Do you have any examples of quick experiments? Yeah. I mean, I think what I, and in fact I'm doing this right now with a client, is that they are working on a new product and they believe they understand what the alternatives are today to that product in the market. That is brought with assumptions, assumptions about how they solve the problem, how their friend who also, you know, works in the same space solves that problem. And it may or may not be true. Who the decision maker is, who has the budget to solve that problem, many, many assumptions that need to be true for this solution to exist. And so for us, we're doing a very quick experiment. And again, people do equate experiment must be an A-B test. Not true. It is how do we get evidence and data that tells us whether the assumption is true or false. And in this case, we're talking to who we believe the buyer is. So we're going to run an experiment to see if we actually know who the buyer decision maker is by seeing if they care to have this problem with us, have this conversation with us. They don't care to have the conversation with us. Either we are not talking about a problem they care about or we pick the wrong buyer decision maker. And two, we're testing how they solve the problem today. We have some beliefs and assumptions about how they solve it. If those are not mentioned, then we know that we do not understand how they solve it today. So those are going to be two experiments that we're running with a single test, which is trying to schedule an interview with these people and seeing if they're willing to talk about the problem and how they solve it today as an example. Yeah, those are great. That's great. So we see a lot of, one of the things that I really encourage teams to think about, both of us do, is when you're running an experiment, how do we define an experiment or a prototype? The key is that you need to simulate the experience the person would have if your product existed, but not the whole product experience. It's just the experience they would have to test that particular assumption. So I'll give another example. I'm working with a team right now who their idea is to build a community forum where people can share pricing. The price they sold, I can't give too many details, but I can say the price at which they sold a commodity good. So they all want to know, they're all constantly asking as they watch the commodity price fluctuate, should I sell today or should I sell tomorrow? So they want to be able to collect regional data about who's selling when and what price they got. to help them make this decision about should I sell now or later. One of their assumptions is, will our customers be willing to share the price at which they sold? Right? So they don't have to build this whole community site to test that. What they're doing is they basically, they want to simulate the experience. They want a customer to say, yes, I will share this data publicly. So what they're doing is they're actually going to write a blog post. Well, they're going to tell their customers they're going to write a blog post about the experience of five customers who recently sold their commodity and ask if the customer would be willing to share the price for that blog post. So they're creating an easier instance in which to share publicly. They already have a blog. They actually don't even need to write the blog post. They're just going to ask the customer if they're willing to participate in the blog. And if they are, to tell them the price. So it's not just, yes, I'm willing to share my price. It's yes, I'm willing to share my price and the price was this. That's something that they can do in a day or two. They can reach out to customers. They can ask for that information. They're either going to get it or they're not. They didn't have to build a single thing, but they're able to test the assumption. Now, it's not 100% perfect, but it's pretty darn close. The blog readers are the same people that would be part of the community. It's not anonymous. They're going to learn pretty quickly about whether it needs to be anonymous or not. So it's something, it's a really teeny tiny activity that allows them to get a lot of data quickly. And I think what you just, what we shared in those are sometimes we're testing desirability assumptions. So the example I gave was really around desirability. Do we know who wants to solve this problem and how much they like or dislike the way they solve the problem today? And what Teresa described was sort of like a customer feasibility problem. Like, are they willing to do it? Are they willing to contribute their price paid information, which would be required as a feasibility assumption for the solution to exist? Yeah, and this is where the categories get a little bit noisy, because I would actually define my example as a desirability assumption. Are they willing to do it? Do they want to? Right? Whereas some people do categorize it as a feasibility assumption, like is it even possible? I think what assumption a category falls into doesn't really matter. I look at the categories as lenses, right? If we were to look at this solution from a desirability standpoint, a feasibility standpoint, a viability standpoint, etc., does that help us generate more assumptions? so that you're better able to deconstruct your idea. So as you work through this, don't split hairs about it. It doesn't matter if it's feasibility or desirability. It just matters that you identified it and you can test it. And I think the other thing that helps you identify assumptions that you should really test soonest is if that assumption needs to be true for any of your solution ideas to work. that those are going to be the ones you're going to want to test it. So in Teresa's example and in my example, like if none of those are true, it doesn't matter how great our solution is. There will be no market for it. So that is a very important thing that we want to test early so that if this turns out to be a two-way decision, we decided we didn't like what we saw on the other side, we pivot right back up the tree and look for another opportunity. Yeah, and I'm noticing in the chat that Ben, It looks like it only went to all panelists. I don't know that everybody could see this, but he's highlighting that he likes that experiment design because it's easy for people to say they will do something, but it doesn't mean they will do it. And that's why we really emphasize when you're prototyping and experimenting, we really want you to simulate the experience so that you're measuring action, not just, yes, I would do that. Because here's the thing. I'm going to eat vegetables every meal tomorrow. And I'm going to go to the gym every day of the week. And I'm going to get all of my work done. Not now you're not. Because that's what I believe in this very moment, right? But tomorrow reality is going to happen. And I'm going to run out of spinach. I'm not going to eat it with my breakfast. And my day is going to be jam-packed. I'm not going to get my expected workout in. Right? So we're really optimistic about what we will do. But so we want to make sure that when we're experimenting, we're simulating. what so that they are required to do the action you need them to do. All right, we dove into a lot already on this topic, which I love, but let's turn to Melissa. Melissa, what kind of questions are we getting? Sure. So one of the questions we had was about the difference between a good assumption and a good hypothesis. Could you kind of clarify between those two points? Yeah, so I think an assumption is a belief, right? So This is where we really run into the limitations of like product teams are not trained scientists. So I'm going to distinguish between these and then maybe hope and I can talk a little bit about what's really required. So I would say an assumption is a belief, right? It's something that we assume to be true. And I think it can be formed in any way you want. When we talk about a hypothesis, now this is a scientific term that has a very specific meaning. And the very specific meaning is that it can be tested. through experiment design. Right? And so in order for an assumption to be tested as a hypothesis, there's a lot of things that we need to do to translate it so that it's falsifiable. And that's really the key that distinguishes an assumption from a hypothesis. So I've written about sort of the criteria I look for for something to be considered a hypothesis. I look for things like Is it specific enough that it's falsifiable? Are you drawing a line in the sand? So are you saying that x people will do y rather than saying people will do y? Because people will do y. I can search infinitely and as long as I find two people, it's true. But that's terrible because I can't search infinitely. So it's not really a falsifiable hypothesis. So when we talk about assumption versus hypothesis, there's this gradient between a belief which can be pretty vague, and something that is falsifiable through experiment, which is a hypothesis. Now a lot of teams, and this is where maybe Hope and I can discuss a little bit about where you need to be, a lot of teams can turn an assumption into a pretty good testable hypothesis really easily. They are natural experiment thinkers. Some teams really struggle. I have an experiment design template. I know There's the, someone I saw in chat, someone highlighted the We Believe format. The We Believe format works really well for some people. For a lot, for other people, it's not specific enough because they don't use it to get specific enough. Hope, do you want to talk a little bit about kind of the gray here and what you see from Teams? Yeah, I think, I think that product teams. Once they're introduced to the sort of creating a hypothesis template that again gives them sort of pass fail criteria that they can use, I think then they can start thinking creatively about how they might scope down that experiment to quickly learn whether it's true or false. I think sometimes the assumption language, especially working with other stakeholder partners can trip people up. is they sometimes don't want to be constrained to that criteria. Because like, oh, well, if you're only going to test it with five people or 10 people in a prototype, and then you draw that it's a false assumption, that might limit our opportunity, or we might be taking an idea out of commission that I still believe in. And so it's super important to be thinking about not just the product team, but who else will need to believe and have confidence in the decision and the action that you're going to take based on how you draw the line in the sand, how you verified whether it was true or it was false, so that you can make sure that you can in fact take the corrected action that you believe. Is it that we're going to iterate on our approach or is it we're rejecting this as an opportunity and never coming back to it? So I think it's important for everybody to be clear about what decisions you're going to make whether that assumption turns out to be true or false based on your experiment design. Yeah, this is awesome. So let's get to a specific example. So if we run with the assumption people will share pricing data. That's an assumption. It's not testable. We can't talk to all people and find one or two people to prove that true. Right? So to turn it into a hypothesis, we might say three out of five customers that we ask will give us pricing data. That's a pretty good hypothesis. It is also viable, right? But it may not be convincing for that assumption depending on your organizational context. For some product teams, three out of five of customers participating might be plenty convincing. For other companies, if you have a strong quantitative culture and three out of five you're going to get pushed back and you're going to debate forever, you might have to say a hundred out of a thousand customers asked. give us pricing data. Now we're running a much larger quantitative experiment. It's going to take a lot longer, but if that's what our organization requires, we need to be moving towards a more and more rigorous experiment. One of the things I like to encourage teams to do is to start really small. Start with that three out of five because if nobody's willing to share data, as long as you chose variation in those five, odds are a hundred out of a thousand are not going to share their data either. We run the risk with small numbers of some false negatives and some false positives, but there's enough product ideas out in the world that I think we can take that risk. Whereas really what we're looking for is, is anybody willing to do this? If they are, now we're willing to take some time to run that bigger, harder, longer experiment. Should we take some questions? Okay, we've got a couple of logistic... poll questions about assumptions. So one is where to put assumptions on the opportunity solution tree. And then the other one is who should be involved in defining assumptions. So I'll tackle the first part. I don't think you should put assumptions on the opportunity solution tree, but there's not a right answer here. In fact, I published a blog post on Product Talk from a product team that shared having a ton of value. from putting assumptions directly on the tree. So if that works for you, by all means do it. I really am reluctant to turn one document into everything. Your solutions are going to have lots of assumptions, and I think it's a little too much information for the tree, but experiment. Do what works for you. What I see teams do is they're... collecting assumptions in like a Miro board or a mural board, some kind of digital whiteboard, so that they can move them around as they learn about them and sort of assess risk. And that's just, and maybe that's in the same document as your opportunity solution tree if you want everything in the same place. But I think they serve different purposes. And then I already... I think the second part of that question, I think I remember. I think the second part of the question was who needs to be involved in generating the assumptions, is that right Melissa, or in coming up with the hypothesis criteria? It's who should be involved in defining assumptions. Yeah, so I think this is, it will depend on your organizational context. You know, typically I find that if it's, if the product team is operating completely autonomously, probably just the product team. If you've got stakeholder partners that are going to need to be that will be impacted by the decisions that you're making, you probably are going to want to make sure that they're participating in your assumption generation activities. And there's many options for how to involve them in this. My personal opinion is that if this is something that is going to be something that is a new product being introduced to the market, then you probably are going to want to have some customer facing teams like sales. If this is a how are we going to make this product easier to use, you may just want to have maybe partners from support or maybe it's just within your product team and there's design and usability emphasis. It really is going to be dependent on the decision you're trying to make. That's why when you know what decision you're going to make when you have truth. about the customer's behavior in your simulation and your experiment, who's going to take action on that truth, whether your assumption turns out to be true or false, getting their buy-in and participation in how you set that decision criteria and what action you're going to take is going to, if you have to convince them after the fact that you designed the experiment in the right way, it will probably be more cumbersome and slower than if you had involved them in the first place. Yeah, one of the things I'll highlight was in part one we talked about collaborative team decisions for across that product trio. So at a minimum when we talk about all six of these principles, which will we were really expecting that at a minimum that trio is participating. And then I think Hope's spot-on like depending on where you are in the product cycle, depending on where you are in discovery, depending on the stakeholder management that's required. you're inviting other folks to help facilitate that process. So we have about 10 minutes left. We are going to continue to take your questions. I want to do one thing really quickly, which is just give a quick recap of all six things that we talked about, assuming I can remember them, and talk to you and share a quick way you can learn more. And then we're going to get right back to your questions. So over the last three-part series, we talked about trio based decision making, so collaborative decision making. We talked about externalizing your thinking through visuals like opportunity solution trees, customer journey mapping, experience mapping. We talked about in this visual here you see at the top being outcome focused, so making sure that your teams are tasked with an outcome and not just delivering outputs. In part two, we talked about discovering opportunities. We talked a lot about doing that through interviews, but you can also do it through customer observations. You can do it through analytics. You can do it through a wide variety of discovery activities. We talked about prioritizing the opportunity space and making sure that you're not just ranking features unlike features against each other, but making strategic product decisions in the opportunity space. And then this time, of course, we dove into compare and contrast decisions, making sure you're choosing amongst a set of options, and then of course surfacing and testing assumptions. I like to capture this whole process in a visual so that people can keep track of where they are and what they're doing and what they do next. That's actually how the opportunity solution tree was born. If you want to learn more about any of these methods or how Hope and I teach them or how we work with teams to help them find the mix and match of discovery methods that work best for them, you can learn more about what we offer at Product Talk. dot org slash learn. I'm not going to spend a lot of time pitching our services. I want to get right back to answering your questions. But if you are interested in learning more, check out that URL. All right, Melissa, what do you have for us? Sure. So we had a couple of questions regarding what tools you might recommend using to map the opportunity solution tree. Yeah, I like Miro and Mural. So they're both digital whiteboard tools. People ask me if one is better than the other. I work with teams that use both. They seem pretty interchangeable in my mind. Before Miro and Mural really existed, I worked a lot with teams in Lucidchart, Draw.io, some even used OmniGraffle. Here's the criteria I would look for. It's visual. You can drag things around so you can make fast updates. Multiple people can edit at the same time. Everybody on the team has access to it. Beyond that, I think most tool questions it's really about finding the right fit for your team. Okay great and we had a question about when you're running your experiments do you feel that it hurts the experiment if you offer to pay your customer or interview to do the test with them? Good question. You want to tackle that Hope? Yeah I think again Every situation is a little bit different. You're going to find there's going to be segments of prospective customers or existing customers that will gladly give you their time because they are passionate about the problem space that you're solving. Maybe they're passionate about you if they're an existing customer. They want to have their opinions heard and you may not compensate them at all. There's going to be other people where more of an incentive is required where it's just like happy. I'd be happy to talk to you about it, but my time is valuable and I want to make sure that this is a good use of time. So I think you have to do whatever works. It may not necessarily even need to be some sort of cash incentive. There may be some other value exchange. There's a lot of companies use customer advisory boards or people have opted in to participate in customer research and there may be, you know. They may not exactly fit who you're looking for. You may have some false starts in terms of who you speak with. But I think you want to make sure that you're talking to the right people that make sense for the problem that you're trying to understand or the opportunity you're trying to understand. And you should use lots of levers to make sure you have those conversations with the right people. Friends and family, incentives. value exchange, you're getting early access to see something that is even enough of an incentive for people to participate in research. So I don't think there's really any downside risk as long as you're structuring your conversations to learn what you need to learn to de-risk what you're trying to achieve. I think one of the things that might be under underlying this question is I know some teams run into problems with this idea of like professional testers on these bigger testing platforms. And I think there's two ways around that. One is you have to write better screeners so that they're not just checking a box saying I'm an accountant, but you're asking a question that exposes whether they truly are an accountant or not. So it's especially on these large recruiting platforms for discovery tools. Think about how to screen out the professional testers in your screening process. And then I think the second tactic you can try is most of those tools allow you to upload your own list so you can actually test with your own customers. That's another way to avoid sort of that person who's sitting at home trying to make a living by just doing a bunch of usability tests. Okay, and shifting gears a little bit, the next question is, how do you deal with a situation where executives or customers come to you with strongly held solutions in mind already? Ah, yes. You know, some of the principles that we talked about can be really helpful. So the first thing I encourage teams to do when somebody comes to you with, hey, we have this great idea, let's do it. I actually recommend you go directly to the whiteboard and start. Story mapping what that idea might look like because most ideas are not good in practice And sometimes we just need to take some time to think through how they might work to realize they fall apart Right, so if you get if you do that with your stakeholder and they come to that Relate realization on your own on their own it gets rid of a lot of those problems Now the opposite can happen to you might you might realize hey, this is actually a really compelling idea So the first thing is take some time to make sure that you're externalizing that you're helping that person externalize their thinking what do i really mean by this idea how might it work so that it's becoming really concrete i think from there it becomes a prioritization decision right so i would use the tree or impact mapping or whatever your externalization your tool for externalizing your thinking to work with the stakeholder and say okay this idea looks great can i ask a few questions I'm supposed to be working on this outcome. Is that still the most important thing? It is? Okay, great. We talked about me working on this target opportunity. Is that still the most important thing? Okay, great. This idea doesn't really address that. Can we table it until we're working on an area where that's more relevant? Right? So helping your stakeholders remember, we already made these strategic decisions. Do we really want to disrupt those? Hope, you want to add anything there? I mean, I... Plus one everything you just said. The other thing is, you know, our stakeholders once you've got an idea you get that sort of rush of Oh, I only see how it could work and and it's also then you don't want to be hit with a big old no in response, right? You're trying to be a good collaborative partner. And so again going back to this concept of surfacing assumptions that must be true for that to succeed. Let's just assume for a minute that this great idea that it's just been presented to you actually relates to your target outcome and opportunity. Then maybe it is one of the solutions that you can compare and contrast again. And there are techniques that you can use to get the assumptions out on the table. One of my favorite de biasing the confidence biasing techniques is a technique called the pre mortem, which essentially says, Okay. The idea exists in the world, however you conceived it to be, but it failed. What went wrong? And getting people to think through what went wrong almost reverse engineers what needs to go right for that solution to succeed and frankly any of your solutions to succeed. And so that kind of gets people back to what assumptions need to be true. How can we de-risk these to know that they are in fact true so that we can find the best solution to get us to our outcome soonest. Yeah, great. Okay, we are right on the hour. So I want to thank both Melissa and Hope for their time. And I will remind everybody that we will be releasing videos of all three parts. If we did not get to your question, we are going to take the most common questions and record some follow-up videos. All videos will be distributed through the Product Talk YouTube channel. So if you want to make sure you don't miss those, be sure to subscribe. Thank you everybody for those of you on the West Coast. Thanks for spending your lunch with us. For the rest of you, thanks for finding time in your day. See you on YouTube.

This is the third webinar in a 3-part series. 
Part 1: https://youtu.be/ihN8rtjWz3g
Part 2: https://youtu.be/G6UlvEJhQC4