Pawel Huryn - Product Discovery. What is it all about? | Product Owner’s Toolbox Conference

Welcome. My name is Paweł Huren. Since 2014, I have been helping organizations build great product and amazing product teams. And I am also one of the 400 PSPO3 worldwide. I'm mentioning it because I will be referring to Scrum in a few places. And today I will tell you about mainly about continuous product discovery, although I will also mention product discovery in general. So how to identify opportunities, how to test the ideas before they are selected for the sprint and why do we need it? So in product, we should be obsessed with the why question. So let me start with the question, why do we need product discovery at all? And before I answer it, let me define the problem that we're trying to solve with product discovery. uh think about the solution so the quote that you see on the screen is a quote from marty kagan from inspired and the truth about product management is that at least half of our ideas are not going to work also marty kagan mentions that strong product teams assume that at least 75 of the ideas won't perform as they hope so this might be a problem because When we look on the traditional diagram, this is Scrum, probably the most popular other framework. So on the right side, we have creating. So, Agile did an amazing, I would say, that Agile did an amazing job in delivering software in small iterations. inspecting and adapting and asking the question are we building the right thing so for example in scrum we have this sprint review meeting in which we can together with the stakeholders we can review the increment review the progress towards the product goal and collaborate on what to do next and this allows us to adjust the direction and react to changes in the business environment but the work that is done in agile or scrum i would call it a delivery work so we take some product backlog items or some ideas they are implemented and then we can verify the result of that implementation so this is building the product and learning by delivering and that way we can deliver value faster we can adjust the direction But if the ideas do not work, it can result still in some amount of waste and rework. And if we assume that at least half of our ideas or 75% of the ideas will not work, implementing them just to find out that those were not good ideas is not the most efficient way to create software or to create products in general. So many people ask different type of questions. So for example, in the Lean Startup, the general product discovery, Eric Rice asked the question, if customer wants our product at all, is there a market for this product? Can we achieve the product market fit if we work in iteration, if we work in agile? So this can be done probably after many iterations, not after the first one or the second one. In Jobs to be Done, Arwig asked about what are the most important problems that we want to solve. And even if our ideas are correct, maybe we should be working on something different. So he was wondering how to prioritize problems that we are solving. Of course, this is not the only topic of this book. And product discovery, I'm still answer the question, what we should build. So the way to do it, in the modern product discovery is to run small experiments in front of channel, the front of funnel to validate ideas as cheaply, as quickly as possible before selecting them for the implementation. So what is selected for the, if we work in Scrum, what is selected for the sprint? We would like the most risky assumptions to be tested before we start working on this. on this item. And this is product discovery. But you may wonder why so many ideas do not work. What is the reason? So Marty Kagan mentions five risks, four risks that product managers should address. This is inspired, but I heard Marty Kagan and also Teresa Torres mentioned. mentioning the fifth risk so they go like this the first one is that the things we create maybe they are not valuable for the customers customers will not desire them and i would argue that in order to make customer switch we cannot have simply good solution our customers need to love the product so that they can choose this over over what they are currently using uh so this is the value risk usability risk this is if they if users can figure out how to use it so maybe we have a great functionality no one knows about or they just don't know how to how to make it work we also Many ideas also do not work for the business and ideas must work for different departments like marketing, sales, finances. There may be many reasons why our business does not support it and product manager needs to make sure that different elements of our business, different departments, different stakeholders can support this idea. It works for the business. There is a feasibility risk. So obviously this is something that engineers can address if we ask the question if the technology allows us to do it. So for example, can we integrate with external system? Can we create a reliable algorithm? Can it be scaled? And so on. And the last question, the fifth that both Marty Kagan and Teresa Torres added later is ETIC. Is there any potential harm connected with our software? Do we have data needed to build these solutions? And maybe customers have some problem with this data? Can we get their permission? So there might be many ethical considerations here. And Jeff Patton and others in the Agile product space, I think... many of you know uh jeff button's work have been big proponents of the approach that is commonly called dual track agile or dual track development so basically um we have two streams one stream is about discovering what to build and the other one is about delivering building the product in my opinion product discovery has it roots in Lean, in particular Lean startup community. So I prefer to use names which also Mwartekagan uses product discovery and product delivery. And the goal is the same. So product discovery is about finding out what to build and product delivery is about building it, learning, inspecting and adapting. So what's Exactly inside the product discovery. There are, I would say that there are four points on my presentation, but there are two groups of activities. So one of the activities, group of activities is exploring the problem space. So we would like to understand the problems, needs, desires. We would like to analyze data, perform customer interviews, synthesize knowledge, perhaps also analyze some uh some reports or some some insights from others and we would like to map opportunities and the second group of activity is exploring the problem space so we would like to identify and test our ideas with the help of prototypes although in a moment i would precise that we are not testing ideas we are testing assumptions corrected to connected to those ideas but basically It's about testing what we need to do. So how it looks in practice. A lot of people repeat that product managers decide what to build and software engineers and designers should focus on how to build it or how to make it pretty. And honestly, it hurts my ears because This is not a task for a single person. This is in line with what Marty Kagan says and also Teresa Torres, that instead of building silos with stage gates, we should embrace a more collaborative approach. So we need to make sure that the designer that is responsible for usability and at least one engineer's engineer takes part in product discovery this will allow us to build a sharing shared understanding and also we can stay open to different perspective because every every person brings different experience to the table and from my experience often the best ideas came from engineers not from designers not from product managers so engineers know what is possible they know how to leverage technology to build a competitive advantage and yeah product manager can understand customers but is not a technical expert so doesn't really know what is what is possible and Using engineers, using designers to work together is, in my opinion, critical. So when we have a product trio, one of the primary tasks is to interview customers regularly. Teresa Torres talks about interviewing at least. customers at least once a week. So this is not something that we do at the beginning of working with the product, but we do constantly in parallel to delivering software. Marty Kagan said in one of the interviews that we should cancel the interview if designer or engineer is not present. And what is important when we start interviewing customers, we should have some business goals in mind, some some outcome that we care about and then we start talking to the customers to to map their problems need desires in short opportunities so this is the green layer that you see on the picture in the picture and what's important is that we do not want to ask customers about what they think about the idea or we don't want to ask them to propose solutions Because people often do not understand their problems. They feel what hurts them, but they are not very good in generalizing. And they also often make decisions influenced by emotions and they try to rationalize them later. So we should be skeptical about what they are saying. and in particular we shouldn't focus on asking customers about solutions. Also, quite important thing is that customers are generalized. So instead of asking about their opinion, I usually ask about specific situations from the past. So, for example, tell me about the last time you had that problem. You can ask how they solved it. If someone was helping them, maybe they found some workaround. If they have not tried to work a workaround, this is for me, it's a good indication that the problem is not important because if this is important, I would try to find some solution, at least workaround. So if they have not tried, that might be a good indication that even if they tell it's important it may might be not and so for example our goal may be to increase engagement by 20 this can be expressed as okr also and one of the problems that we identify may be that i can't find anything to watch but when we when we keep asking why uh the reason may be that they cannot find figure out how to watch to or search for a specific show or that they are out of episodes and usually it's not one or two layers it's more like four five maybe six and after we identify the root cause of the problem we can start thinking about solutions what's important is that we do not ideate together with the customers. So what we really want to know from the customers is how satisfied they are with their current solutions and how important those problems are for them and why they are important, how they influence their life, their goals, personal life. What would change for them if we solve it? Anything that will let us assess if solving this problem can drive the outcome, the business outcome that we care about. And the idea, I think, is something that we do separately, not with the customers. So, for example, for this problem, display a search box. I can't figure out how to search for a specific show. solution maybe to display a search box or maybe display recommendations. And after we have a set of ideas, we can start testing them. To be precise, we don't want to test ideas, but we would like to test assumptions connected to the ideas. And for those assumptions, we need to perform experiments. I will show you in a moment how we can do it. So experiments. test assumptions connected to ideas. This is an example of the customer interview, so quick facts, insights, opportunities. I would skip this one. You can access my presentation later. So what's important is testing assumptions. So the assumption may be that solving the problem will drive the expected business result or addressing target opportunity target opportunity is is the problem is a different name for the problem uh need or desired and it will drive the expected outcome so in our case it will increase the customer engagement because and here is our assumption that should be testable um and instead of instead of just brainstorming about assumptions, there is a better tool that we can use. And it's a story map. So instead of just sitting with a piece of paper and writing what we would like to test, we can create a story map. So in order to do that, we need to map steps that each actor, so our user. has to take to get the value from the solution. And we would like to sequence the steps horizontally over time. So a very simplified version, it's in practice, it will be a little bit bigger. So for example, the first step may be that users go to the movies catalog, then system will display recommendations and user selects recommended movie. So this is a very simple story map. For this story map, we can start thinking about assumptions from different categories related to the risk that I mentioned at the beginning. So value or designability, the assumptions for the first step may be that the user wants to watch movies at all or they want to watch movies on our platform. This is something that we can test. um because maybe they want to watch movies on youtube not on netflix and for select recommended movie the assumption may be that the user wants to use recommendations and similarly we can think about assumptions for usability for feasibility sometimes you can use story map to generate assumptions related to viability so what our business can support but I usually treat it as a separate category because it is not always related directly to the story map. So this feasibility needs to be addressed separately. And of course, we can also think about some risks related to ethical considerations. So this is a simplified picture. And once we generate... once we generate the list of assumptions for our idea like displaying a search box so we may have for example seven assumptions we do not want to test every hypothesis it would be wasteful there is a large number of hypotheses you can easily generate for every idea so what you see on the picture is Hypothesis prioritization canvas. I will share the link to the original source where you can download it. And Jeff got health divided hypothesis into four areas. So the horizontal, in the horizontal line, we have low risk to high risk. And the second, One is low perceived value or high perceived value. And the sweet spot for us is testing those assumptions that are risky, so may turn out to be wrong, and that are connected to the high perceived value, may produce high perceived value for us. If there is a high risk and low value, we can just discard this assumption because there is no No point in implementing something that is risky and won't produce much value. If there is low risk and high perceived value, we can just ship the solution. We also don't need to test it. And if it's low risk, low perceived value, we really don't care. And usually we don't build it because there are more important things on the agenda. Here you have, I will speed up a little bit. Here you see the... So what is important when we create this opportunity solution tree is that we do not want to prioritize ideas like displaying a search box, but we would like to prioritize and focus on customer problems. So it means that we need to understand how important is this problem for the customer, how satisfied they are with the current solution, and if the level of satisfaction is low and they are not satisfied. So it's an important problem for them. It's a good indicator that this is something that we should address first. Also, there is a common trap. It is called a product dead cycle trap. And it is when the project manager forgets about the why and becomes a waiter. So to please everyone, product manager just take requirements from customers, from stakeholders and waterfalls them down to the team. There is no vision, no strategy. And basically it's just a feature factory. So to... Sorry to interrupt. I'm assuming the background was from you. It's not from me. okay let me just see because uh i only saw it i i also hear it but it's not not from me uh let me just quickly fix that uh i'm gonna mute also you might have to unmute and then we'll continue okay so if you unmute you should be up okay sorry i turn now is it better now yeah yeah okay about that i wasn't sure because i saw i was trying to validate but and see if it's coming from but please continue yeah thank you for uh yeah it was annoying so yeah so so to escape this trap we need to think in the long term so um what understand why we are building this product who is our target customer what are the and their self needs that we should really target and Also, in order to escape this trap or avoid it, we really need to understand the problems. And it's also, it might be sometimes problematic because sometimes customers or stakeholders came with some idea and just orders us to implement it, in particular, some powerful stakeholders. But in that case, I would ask. just ask why it is important and what are the success criteria to understand the reason of implementing this feature. And after understanding it, I would try to reform or reverse engineer it so that we define the problem and the solution that the stakeholder proposed is only one of the possible solutions, but we agree with the stakeholder on the acceptance criteria. And it's important to respect stakeholders, but the authority shouldn't influence what is important. So sometimes it's just necessary to question solutions and push back things that are hand-led down to you. OK, so we have I have presented solution opportunity three. So mapping customer problems related to that's goal that we care about then generating ideas together with engineer and designer and identifying assumptions for those ideas that we can test here we know that we should prioritize problems not solutions that are on the bottom of of the opportunity solution tree so now how we can test What are the types of experiments that we can perform? And I will once again refer to Marty Kagan. So Marty Kagan listed four types of experiments. Low fidelity prototype, user prototypes. So this is something that I used in the past. I used Balsamiq. So those type of prototypes can be done really fast, like in hours or sometimes even under one hour and it doesn't look pretty but that way you can validate some assumptions about value the other type of experiment the other type of prototypes is high fidelity user prototypes so high fidelity user prototypes you can use figmas catch adobe and it aims to test also the usability part in more detail, including UI, not only some flows and interactions. Another type of prototype that you can sometimes see, for example, in extreme programming, are feasibility prototypes, like, for example, spikes. So basically, this is some work that needs to be done by developers to address the feasibility risk, like some algorithm, some scalability issue, integrating with library, integrating with the external system. And I think most of us have seen it at least once. And live data prototypes, this is the most important, the most advanced type of experiment. It basically requires implementing some part of functionality, but this is not a real product. And it is often used in A-B testing or invite-only testing. So this type of application doesn't have to support a lot of users. In A-B testing, we can just say that 1% of users will see a new version of the application. It is not intended to implement all the possible scenarios, maybe only the simplest one. We can skip advanced testing, we can skip scalability issues, we can skip internationalization, so it can be only in English, for example. Also, performance is not that important. We just want to test some part of application, so create this prototype, but it must be working with the real production data so that we can generate real data and then use analytics, compare the current product with the new version. Okay, and now you may wonder, this is the reference to Scrum that I would like to do because most of us... I assume work or have worked in Scrum in the past. So how to combine continuous product discovery with the Scrum framework? Let me present you a few quotes. So I would make a distinction between the initial product discovery before we have a product that is well defined. And according to the Scrum guide, we have the product has clear boundary. has known stakeholders, were defined users or customers. So if we are just starting working with the product idea, like in the Lean Startup, Scrum may not be the best solution because it simply requires to have the product defined. And in the initial product discovery, we don't even know who are our customers and we are trying to to define value proposition and test our idea often with the mvp prototype so initial product discovery aside and for the continuous product discovery this is the time when we have a product that has boundary has known stakeholders has well-defined users and customers and in that case Continuous product discovery should run in parallel to product delivery and we interview customers regularly on Ideali every week and it allows us to validate our ideas, at least the most risky assumptions before they are selected for the sprint. And here I collected artifacts of the product discovery and of product delivery. They are different, but I think I will skip it. You can read it later. It is just to show that product discovery and product delivery are two different things. And you already have seen that picture. So if we connect the dots, product discovery generate ideas for the product backlog. Some of them may be validated, although I do not fully agree with this quote from Marty Kagan that product discovery results in a validated product backlog because some ideas which are not risky can be directly implemented and this is also supported by Teresa Torres. take some low risk ideas and we can ship them and then learn by after delivering product discovery as it's a little bit messy it's i would say that it is more dynamic so it's hard to have this constant rhythm but it consumes a similar amount of work every week so if you care about story points about velocity it shouldn't affect it so it should be similar every week and you can put all those draft ideas in the product backlog but i would say that you don't have to do it so for example um if there is a work connected to interviewing a customer for me it doesn't make sense to put it in the product backlog because this is something totally different and it would keep our notes on some Confluence page or we will have some meeting in the calendar and making a product backlog item only because we need to interview the customer. It's not related, really not related with the sprint goal. This is something that is done in parallel but if you want you can implement some workflow of product backlog items and have some draft states for the ideas that needs to be validated. So some recommended materials. I will share this presentation on this page that was visible at the beginning. So Opportunity Solution Tree, a great article by Teresa Torres, the product that cycle trap article about this trap. uh silicon valley product group on prototypes and four types of prototypes this is this has been written by marty kagan hypothesis prioritization canvas that was visible you can download it for free uh teresa torah's video about introduction to modern product discovery and what's interesting i mentioned jobs to be done by tony ulwick and this book can be downloaded for free as i an ebook or an audiobook so um yeah it's it's interesting that that you can get it for free in a in a digital form Also, I collected some recommended books, so Inspired, the best book for product managers, in my opinion, and it discusses the principles of product discovery and the truth about the product management that most of the ideas will not work. The mom test. So to say it shortly, you shouldn't ask your mom if you're business idea is a good idea because she loves you and she will lie to you so i am simplifying but this is this is the the the uh conclusion from the book continuous discovery habits number one book about continuous product discovery build trap by melissa perry It's about why we shouldn't let our customers design solutions and why we should focus on outcomes, not on the output. And Radical Focus by Kristina Wodtke. This is a book about OKRs and why OKRs are a strategic tool and why there should be only one OKR for every team and for every business model. So it's really interesting. i will also paste a link to so this is not connected directly to the presentation uh partially the biggest collection of product manager learning resources so books podcasts videos internships free courses conferences for the next year framework blocks and much more and i keep this list updated so you can find it easily find it and for example look for the internship or look for some free product management course do you have any questions so maybe Paul just quickly before we jump into could you give me uh maybe the links that you share and also people have asked me privately about presentation so if you could if you're okay which send me the email and then the link where they can get these because I think Paul is one of the people that has like such so many great resources if you're not following paul follow him uh he posts regularly it has some really good stuff but uh maybe you know put in the chat any links that are quickly and i'll put them in helpful resources uh and yeah let's open up for any other questions that people might have so one of the questions from kyle i don't know if you can see but how I don't see it. Okay. No, that's not a question. Sorry, it's more observations. There's Shaif is asking, how can we learn about validation experiments? So how can we learn? So maybe if I understand the question, so how can we learn? Shaif, maybe could you go off mute and just... Yeah. Hi, Pavel. This is Shif Ali. So my question would be like, is there any... consolidated way to learn about the various type of validation experiments that we can or it like i know it is it depends on the product to product as well and industry to industry but is is there like in the product management like whatever resources and tools and learning we have there is not much material on the validation experiment Yeah, and to be honest, Inspired Martikagan describes different types of prototypes, but you can learn about it. But this is not a full list of everything that can be done to validate the business idea. And I think I would recommend testing business ideas by Alexander Osterwalder and I forgot other people. So yeah, testing business ideas by the strategizer. This is the most comprehensive book on this topic, in my opinion. Okay. So different types of experiments and not only MVPs are described there in detail. Okay. Another question. I think I answered the one where you had an example of 20% increase by 20%. That was just an example. of an outcome. Another question is where can we read? Sorry, it should be outcome that the team can influence. So it shouldn't be an outcome like increase revenue or even decrease the charge rate because it depends on not only it doesn't depend on a single team. It's also marketing, customer success, support that can influence so it's important that the team works on the goal that they can influence great um yeah I think the scope is always something uh how much influence do we have uh Satya is asking where can we read more about dual track agile right now I don't have a a good source for it but I guess you can find some articles on medium I can I can collect uh recommended materials yeah and maybe send it to me and I'll add it to the helpful yeah I will include it okay uh one more question here uh from Ashish is what can be done if we don't use prints and startups um so if you don't use prints I think it doesn't influence product discovery, initial product discovery or continuous product discovery at all. You still will have some backlogs, the list of works, things that needs to be done, some list of ideas that needs to be implemented. And yeah, you can use flow and some lean practices just to do it continuously. but you still need to to interview your customers you still need to map opportunities so problems needs desires you still need to validate ideas before selecting them for the implementation so i don't see a big difference from that product discovery perspective yeah okay that's it uh as far as i see questions uh in the chat abio doom and I'm not sure if I'm pronouncing your name correctly. You have your hand up. I don't know if you have a question or... That's absolutely spot on. Thank you, Nijak. And great presentation so far. So I think I just wanted to point to something and I'd like you to just make that distinction. What is the difference between assumptions and ideas? And in your presentation, I think you did note that do not validate ideas. what validates assumptions. I know that sometimes there's that thin line between both of them. So the assumption... So the relationship between ideas and assumptions, it is not one-to-one, it's not one-to-many, so some assumptions may be common for more than one idea. And the assumption is something that We can test. So as I presented, let me go back to a few slides. It won't be pretty, but here we are. So it was a user story map. So for example, an idea may be to display a search box, display recommendations, right? And for this idea, we have identified five, six. assumptions. So the assumptions for displaying recommendation idea is that user wants to watch movies and if they don't want to watch movies or don't want to watch movies on our platform, then this idea is not correct. But this assumption may also be important for other ideas that we have on our opportunity solution tree. And some assumptions may be specific only for this idea of displaying recommendations. So, for example, our technology allows creating a reliable recommendation algorithm. This is specific probably only for this single idea.

Product Owner’s Toolbox Conference
Pawel Huryn - Product Discovery. What is it all about?
Agile to agility