🟡 Pawel Huryn - Continuous Product Discovery

Okay, so thanks for the invitation. My name is Paweł Kuren. I've been working for more than nine years as a product manager, including five years of running a startup in Poland. And today I would like to tell you a bit about continuous product discovery, which is the most important area for a product manager of an existing product. Let's see if it works. Okay, but before we discuss what product discovery is, and how to combine it with continuous delivery? Okay. I would like to define why do we need product discovery because as product managers we should be obsessed with the why question at least during interviews. So is there anyone who has ideas that always work? I would like to meet that person. So the simple truth in product management is that most of the ideas Sorry, I need to calibrate this. Okay. So the simple truth is that most of the ideas are not going to work. And in Inspired, Marty Kagan even said that at least three quarters of our ideas won't perform as we hope. So why does it happen? So there are five risk areas. Value. So it means that the solution, our ideas, do not create value for the customers. Usability is about... Whether customers can figure out how to use our solution, can they find the feature? Viability is about our solution working for different parts of the business like marketing, sales, customer success, finances. Feasibility is about what is technically possible. So can we integrate with the external system? Can we create an efficient algorithm? And last but not least is ethic, which is Should we build it at all? Are there any ethical considerations? What will be the consequences for the society? What will be the consequences for customers using our product? And I would argue that product management is in its heart about managing those risks because in practice, in most cases, one of those risks materializes. Okay, you may know this diagram. This is Scrum framework. So I must say that agile did an amazing job when it came to delivering software in small iterations, inspecting, adapting and asking are we building the right thing? So for example in Scrum we have the sprint review which is a workshop. This is not a presentation so together with stakeholders we can inspect the increment and decide on further adaptations. But what will happen if we start throwing random ideas? into the product backlog. So some ideas, as you see in the picture, may work, but most of the ideas are simply not good ideas. And this is agile learning by delivering. It creates a lot of waste, results in a lot of rework. And some might say that, yeah, Paweł, but does it really matter? The sprint is like one or two weeks. Yeah, but it will happen every sprint. So every sprint we select some ideas for the implementation, developer implement them, we test those ideas, we organize a workshop or even better test those ideas with the customers using our product for real and it turns out that most of the ideas do not work. So there must be a better way. Another problem with this when product manager just put some random ideas to the product backlog is that the best ideas might not even be on the list. So the question is how can we came up with better ideas and how can we validate those ideas before the implementation so that the risks related to value, usability, feasibility, viability and ethic are addressed before the implementation. So that when we select some ideas, select some features for the sprint, high risk assumptions are already validated. And the answer is continuous product discovery. So Jeff Patton and others in the agile product space have been proponents of the approach called dual track agile or dual track delivery. So the goal of product discovery, so those are two streams that run in parallel and the goal of product discovery is discovering ideas to build and validating the risks and product delivery aims to deliver those ideas to the customers. And what is important here is that this is not a waterfall. So every sprint, every week, we talk to the customers, we ideate, we try to understand the problems that our customers have. And the same team works on delivering ideas and testing ideas. that are already delivered. And there is this common misconception. Some say that product manager decides on what to build, and engineers and designers, they just make it work and designers make it prettier. Have you heard that before? Yeah? It hurts my ears because product discovery is not a task for a single person. And I deeply believe that we should embrace the collaborative approach so that we should make sure that at least one designer, one engineer and product manager works together because they have different perspectives, different experience. They bring this different experience to the table. And if we believe that customers do not know how to solve their problems, why so many people believe that product manager should know it? Product manager is not a technology expert, just like the customer. Product manager may be tech literate, but I have repeatedly found that the best ideas often came from engineers because engineers are experts in the enabling technology. Engineers have the best knowledge to figure out how to harness technology for the competitive advantage. And this concept of product manager working together with designer, working together with lead engineer, this is called product trio. And this was introduced by Teresa Torres in Continuous Discovery Habits. So what those people do? Inside product discovery there are two groups of activities. One is discovering the problem space and another is discovering the solution space. As part of discovering the problem space we would like to interview customers to understand their needs, goals, desires, gains. But we can also leverage other sources like product analytics, like interviewing internal stakeholders, like analyzing the market, doing some market research. And we would like to structure our knowledge about the problems. And the second part is solution discovery, where we ideate, we think about how we can solve those problems. We identify assumptions that can be tested. And then we perform experiments to validate our assumptions so that risks related to value, usability, feasibility, viability, and ethic are addressed before the implementation. Not all of them, but high-risk assumptions should be tested. And how to do it in practice? This is something that many people struggle with because there are different approaches, and I would argue that none of the approach is complete. So we will start with opportunity solution tree, but then I will present you how you can combine it with additional techniques. Okay. Okay. So we start with the product goal or the desired outcome that is at the top. So this has opportunity solution tree has this hierarchical structure and the product goal. This is something that the product team can directly influence so this shouldn't be about revenue this shouldn't be about customer churn the product outcome this is an outcome that is related to change in customer behavior and yeah that team can directly influence so for example if the business goal desired business outcome is to reduce churn then we may translate it to some product goal, product outcome that the team can directly influence like increase engagement by 20% and this may be also expressed as objective key result. And after this goal is established, you would like to interview customers and understand what are the problems that our customers have. Then when solved will drive the expected metric. So what we can do, what problems our customers have that when solved will result in. increase engagement. And in this case, let's assume this is a video streaming platform. So the way to increase engagement during interviewing customers, we can figure out that one of the problems customer have is that they cannot find anything to watch. And if we dive deeper, it might turn out that they cannot find for a specific show. And the next step after we structure our knowledge about customer problems, about opportunities, is to think about possible solutions. So for example, in this case, display a search box, can figure out how to search for a specific show. The solution may be to display a search box or display recommendations. And finally, the last step is testing. So we would like to think what needs to be true for this idea to work. What are the assumptions related to value, usability, feasibility, viability and ethic? And we would like to perform experiments to test those assumptions. And you can do it in Myro, you can do it in PowerPoint, you can do it even on paper. But I would like to present you a set of templates that I developed. And every participant of this presentation can access them for free. I will share the link at the end of the presentation. Okay, so the first step is to perform customer interviews. So this is a template in Notion where we can document our interviews with the customers. So what is important here is to note some facts, note memorable quotes from our interviews with the customers and also note potential opportunities and how important those opportunities are for the customers, how they are satisfied with what they already have. But what many people miss is that they limit the product discovery work to customer interviews and there are also other sources. So one of them is product analytics. So it is essential in my opinion to leverage product analytics and understand what customers are doing across the customer journey and then by interviewing customers. you can understand why exactly they are doing what they are doing. Another source is interviewing your internal stakeholders like sales, like marketing, like customer success. Of course, they may be biased, but those people spend hundreds or sometimes thousands of hours with the customers regularly. So ignoring what they know, ignoring those insights and starting from scratch, in my opinion, this is a waste. This is a common problem when a new product manager starts in a new organization and then even without talking with the people that are already in the organization, thus starts interviewing customers from scratch. Before they can discover the problems, before they can understand what's going on, it may take a lot of time. I don't like taking interviews too far, so typically I would perform. 15, 20 interviews, max, and many problems are obvious. If you are in a more unexplored space, you can increase that number. So the next step is defining how we can solve those problems. And this is the list of opportunities. So at the top, we have a specific customer problem. And then we can we have customer problems and sub problems and for some reason slides are changing okay so once we understand customers understand the problems that they have we would like to think prioritize opportunities because we can discover dozens of different opportunities and we don't necessarily want to pursue each of them and the way to do it the recommended way i am using is by using opportunity score by the Nelson so this has a very nice graphical representation so the more important the more important a specific problem is for the customer the more value we can create and also the less satisfied customers are with what they already have the more value we can create by solving a specific problem So this case a nice visual representation this yellow area I Compared if we've jobs to be done opportunity score which has a different formula and Where it matters the most it gave similar results. So that's why I prefer dance approach And for a for every opportunity the next step is to think about possible solutions So we would like to ideate, we would like to think how we can solve those problems. And Teresa Torres recommends brainstorming individually and then meeting to aggregate the results. If you need some number, I would typically think about 20 different ideas for a specific problem. The more, the better. And when considering different ideas, we should... take into account not only how important the problem is for the customers, not only how satisfied they are, but also what is the cost of implementation. Because some ideas, some problems customers have may have a great potential to create value for the customers, but they may not be viable for the business. So that's why I estimate every idea. It may be a Fibonacci number, it may be on a scale from 1 to 10. This is an abstract number and then you may want to pursue ideas for which the opportunity score divided by the estimate is the highest. Okay, once we have ideas you would like to think about what needs to be true for this idea to work. So let's say we have this idea of displaying recommendations for a video streaming platform and the way to identify many of the assumptions is by using user story map. So this is like the list of steps the customer has to take in order to get value from the solution and the horizontal lines are different risk areas. So for example, the first step maybe go to the movie catalog and we can think about usability. We can think about value. We can think about some technical assumptions. So for example, for this first step, go to the movie catalog. Our assumption may be that user wants to go to the movie catalog or another assumption, maybe that we believe that user wants to browse movies on our platform, not on YouTube, for example. Maybe they have a different goal. For the next step, display recommendations, we can identify assumptions related to feasibility. So this means that, for example, our technology allows us to create a reliable algorithm so we can actually display those recommendations in a specific time so that user can wait. And all those assumptions related to values, ability, feasibility and viability. This is the list of our hypothesis. So things that needs to be true. And of course, we do not want to test every hypothesis. The factors that suggest testing hypothesis is that we can do it cheaply so that the cost of experiment is low, that we can do it quickly because if we need to perform an experiment that will take two or three weeks, for example, it might be easier just to implement the idea and test it on production. We would also like to test hypotheses that involve high risk and high risk also is related to not only for risk for the product or risk how customers will react, but also there is related to how difficult it is to implement because we risk money. And what is not visible on the screen is that this list combines strategizer testing and learning cards. So this is the recommended approach I'm taking. So for every hypothesis, we define what we believe, we define the metric. we define what is the expected value of this metric and how would we know that our hypothesis is correct and we do it before running an experiment. And after the experiment, we can document our learnings and plan the next steps. Okay, so those are very simple concepts. So let's summarize them. So the first thing to remember is that most of the ideas do not work. And just because we are using Scrum or any other Azure framework, it completely doesn't solve the problem. Because even if we work in short iterations, in every iteration, most of the ideas are not going to work. And also, if product manager manages the product backlog alone, the best ideas may not even be on the list. The goal of product discovery is to create a validated product backlog. So it doesn't mean, it doesn't guarantee that every idea will work, but the high-risk assumptions should be addressed before the implementation. It is essential that the product trio works together because product manager is not a technology expert, product manager is not a UX designer, and especially if we include designer. After ideas are developed, it's like lip-sticking a pig. So the design must be present from the start and similarly technology, similarly the technical competences should be present to leverage the technology for the competitive advantage. It is important that we do not want to apply prioritization to ideas. We want to prioritize problems our customers have, prioritize problems that when solved will create the most value for the customers. But we also need to account for the economical perspective. So if implementing the idea is too expensive, it might not be viable for the business. And last, we would like to validate assumptions, not ideas. So when we run the experiment, For example, this search box or when we have the recommendations algorithm, we would like to test specific assumptions, not just ideas as a whole. The thing that I didn't mention is that when interviewing customers, we shouldn't rely on people's opinions because people are biased and when we ask them, so the way to, so first when interviewing customers, We shouldn't ask them about what they think about their behavior or what they will do. It is way better to ask about specific situations from the past and operate on facts because then it's much harder to be biased when we are asked about the last time we had a specific problem and how we solved that problem, if someone helped us, if we used some other tools. And similarly when testing ideas we do not want to just to present an idea to the customer and ask what do you think about my idea? Would you buy it or would you use it? Is it pretty? So those are not good questions. My favorite approach is automating experiments with tools like Mace and giving people some tasks to accomplish or there are many experiments I also describe. in my newsletter, like a five second test when we present a prototype for a very short period of time. We can also, especially at the earlier stage, we can make a cognitive work through or think aloud, which is that we present a prototype to the customer and then customers explain their thought process as they interact with the prototype. So this is a great way. to identify any usability issues and any issues related to how customers perceive our solution. Okay, so I described this template and described continuous product discovery in much more detail in continuous product discovery masterclass. And on the screen you can see 100% discount code. So it's 100% free without any additional conditions. The one thing is that I don't want to share it with everyone, so it needs to be redeemed within 48 hours. I assume that maybe this recording may be on YouTube or somewhere else. So this is 100 discount code for a video course and certificate. Yeah, normally it costs $149. So in the course I explain how to combine top product discovery techniques like opportunity solution tree, opportunity score by then also an strategizer testing and learning card. You will get an access to a dedicated template that I presented. This is a notion template and there are digital verifiable credentials. You can easily demonstrate after passing the exam and exam is, it is doable and you can try it a few times. I am not sure this is the valid link because I was using the presentation. It was not my most recent presentation for some reason, but yeah, if it doesn't work, I will share the link in the Discord and also you can mail me. So in theory, you can download the presentation by using the link at the top and the QR code visible on the screen. This also includes the previous slide with the link to continuous product discovery masterclass. I also encourage you to subscribe to my newsletter where you can every week get one actionable tip, resources, knowledge, templates for product managers. And if you wanted to subscribe to my newsletter as a premium subscriber, there is a secret discount code also valid for 48 hours. Okay. Do you have any questions? Thank you. Thank you, Pavel. A big applause, please. Thank you very much. We have this column between the stage and the computer, so we are improving for the issue with the controller. And thank you for your feedback anyway. So we have some questions here. What's on the left side of the general problem in the problem space? Any advice about how to explore that? This was a double diamond. And on the left side we have exploration. So exploration is about collecting information without structuring this information. And this includes interviewing customers, interviewing stakeholders like marketing, sales, success. You can make the market research. For example, see what are the market trends and yeah. What is your competitive environment? And then the left part of the left part was about exploration and the right part, so the second step in the left part was about structuring our knowledge about problems, structuring our knowledge about opportunities, and then we can continue with discovering the solution space. The second one is which tool do you use to collect the content of interviews? So I usually do it in Notion, but there are also tools to record and transcribe interviews automatically. So even if I do it in Notion, I try to record an interview in Teams, on Zoom, but there are better tools. Talking with people. But yeah, there are better tools that allow you to index and automatically search through a set of interviews. I was sharing the name of the tool, but I don't use it, so I don't remember it now. This one. I really like this approach. Thank you, Dario. It looks very simple and yet comprehensive. Is there any friction in implementing this approach in practice? It is very simple. Yeah, the friction might be within the organization. So sometimes people say that they don't have time for discovery and this is a common argument and maybe not from product managers but also from the leadership. So in this situation I would ask if they are happy with the current outcomes. So if customers are happy, if what they deliver works for the business. They don't need to change anything. But if not, maybe they should consider making an experiment. And for example, for one team or for one product within the company, try a different approach. Because product discovery and continuous product discovery is ultimately about saving time and saving money. So this is not something that you do additionally. Another one. Do you ever rule to move the hypothesis in testing phase? Are there cases where testing is not recommended? Yeah, as I said, you don't have to test everything and it doesn't make sense to test everything. So if the risk is low, it doesn't make sense to perform an experiment. And also if, yeah, sometimes you can experiment directly on production, especially with A-B testing and some simple changes. Another case when you don't need to experiment is If you have an urgent issue on production, so in that case there is no time to test. You just need to fix the issue because whatever you deploy, it will be better than the software that is not working at all, for example. We have a lot of questions here. Do you think prioritizing opportunities can be the same as prioritizing risk? What do you mean by... Maybe Romeo can you explain it? Opportunities are different than risks. We don't really prioritize risk. We can assess the risk related to a specific hypothesis. So the risk is it has a value the probability and the impact and yeah so we can also multiply but you do a different type of multiplication. Okay. What are the best experiments you can perform to validate your assumptions? So I really like experiments that do not involve talking to people because when interviewing customers it's nice to meet face to face but then when you verify your hypothesis interviews might be more biased and what I am using a lot is a Maze platform to automate my experiments so Those are experiments like you can import a clickable prototype like Figma and create a series of steps that user is supposed to perform. And you can measure misclicks, you can measure time spent on every screen. You can then follow up with a close-ended or open-ended survey. So those kinds of experiments also when working with landing pages or working with new models, you can present a prototype for five seconds and then ask people what they can remember and what the screen was about to verify the messaging, verify if the interface is easy to understand. Another type of experiment is first click testing. So you present a prototype and ask people to for example to make an order and then you verify if they click the right button. Okay, thank you. Yeah, and this is a good question how to find time for product discovery? Yeah, I would argue that you don't have time to not to do product discovery because ultimately this is maybe plus plus for developers for designers this may be like plus 20% of the time plus 10% of the time. But it allows you to save 50% of the ideas that do not work. So of course it is more time for product manager, but ultimately for the team this is a great time saving. And also product discovery allows you to discover ideas that you may not have discovered otherwise. I think that if you have no time to do product discovery, maybe you have not to deliver your product. No? Anyway, can we use the same techniques to validate a new product? This is a very good question. For a new product, we do it typically differently. So you still have this diamond with exploring the... problem space and exploring the solution space. So in approaches like jobs to be done, you still want to understand what are the problems customer have or what are the jobs customers have, how important those problems are, how satisfied customers are with what they already have. You will typically do a lot more, a little bit more market research. But the most, the biggest difference is how you test your ideas because in continuous product discovery you test every idea separately and for initial product discovery the first thing you want to test is your how market will engage with your product idea so you do not necessarily start with testing a specific features or adjusting the button but it's more like using and this is the term that many people may disagree with using MVP prototype which is as defined by Dan Olsen, this is not a product, this is just an experiment so that you can learn and validate your hypothesis. Alberto Savoia calls it prototyping. So it's about pretending you have a product. But the goal, the point is that you would like to verify how market... Engages with your idea you would like to test your messaging you would like to test your value proposition before you build anything and then when you confirm that It might be a good idea Then you can follow up with more detailed experiments like testing values ability feasibility viability for different features and In my newsletter there is a free article an interview with Alberto Savoia when we discuss concepts like skin and gain points which is about a real customer commitment so not about customers that they say that they like you your idea but they make a real commitment like money or time or even email address that proves that they are really interested and another concept that is good to know is xyz hypothesis which is how you can express the market engagement hypothesis so it is measurable and you can design an experiment to validate this hypothesis there is another good question how to start product discovery in the organization yeah so so hard there are two cases one is that the organization wants to start product discovery so in that case i would encourage them to start with one product and one team just to so that they can later share knowledge in the organization because even though product discovery is a good technique, yeah, it still can be implemented in the wrong way. So you need to start learning gradually and similarly if the organization is against then I would look for arguments why we need to change something. So for example, customers are not happy or maybe we have a big chair or maybe a lot of features in our product are not used. So find some arguments, find some data, some statistics, maybe customer surveys, and then use it as an argument to change something, to experiment. And again, this is the same approach. One product and one team and then If it succeeds, then we continue with the rest of the organization. So the last question, how do you choose the right people for product trio? What are you searching for? You're searching off. So you need at least three people. One is product manager, one is product designer, and one is engineer. And for the engineer, it is you. Yeah, for me it is, at least in practice, it's good when it is a senior person. I don't really like rotating people like every week a different developer takes part in this, at least in interviewing customers. Of course, it doesn't mean that we shouldn't include others when discussing solutions, discussing problems, we can invite them for the interviews. But what works for me best is having a single point of contact. which is typically the most senior person in the development team. And also designer is good that it's not a graphic designer, but the person who understands user experience and yeah. So this is not a person who just makes things prettier, but actually understand the usability perspective. Thank you very much, Pavel. A big applause, please.

🎙️ Pawel Huryn - Keynote Speaker | Author, Product Coach & Product Manager
🇬🇧 Talk in lingua inglese

Discover and validate the best product ideas. Prioritize opportunities to maximize the outcomes. Apply best practices and avoid common mistakes. Easily identify hidden assumptions and plan experiments.

With this Keynote you will learn how to combine top product discovery techniques: Opportunity Solution Tree, Opportunity Score and Strategyzer cards (©Paweł Huryn).