"Continuous Discovery Habits" book review/ discussion with author Teresa Torres and Product Managers

Yeah, welcome everyone. My name is Carlos and this is ProductBookLab.com where we meet every month to discuss a book related to product management. And today we're going to discuss Continuous Discovery Habits by Teresa Torres. And we have Teresa with us in the call. Thanks for joining us. Today we wanted to try something new, so I will share now a link. on the chat where you can write the questions and then you can also vote for the questions so then we we make sure we we pick the ones that maybe are more relevant for the group um so yeah you will anyways be able to make the question and then yeah i think we we can start uh maybe teresa we start with a quick introduction about yourself i guess everyone will know you but maybe good to give a quick intro anyways Yeah, so I'm Teresa Torres. I work as a product discovery coach. I've been doing that for the last eight to 10 years. There was a little stint at a, I took a full-time role for about 13 months in the early side of that. And I've been really lucky. I've gotten to work with teams all over the world in a wide variety of contexts. And my book, Continuous Discovery Habits, is really a summary of sort of the tools and tactics that we've... collaboratively evolved together which has been a lot of fun nice yeah it's a very nice book i think now i have it on my desk and for always use maybe to kick start that discussion i will start with one uh yeah question maybe i had for you on the i think of course a very important part of the discovery is as you mentioned then when you have the product outcomes then you want to make sure that you test that they also push and move the business outcome at the end of the day right and how do you see this working when for example there are maybe two or more product teams that they all have different product outcomes right so then it becomes maybe a bit more difficult to measure if indeed this long term is also moving that business outcome, right? Because I think on the on the book, the examples where you mentioned one thing was doing this and then long term they saw that indeed it moved. But I was thinking like when there are, you know, different product teams that, for example, might be contributing to that same business outcome. Do you have any techniques or maybe tips on how we can try to measure or validate that all of these ones indeed move that business outcome? Yeah. So it kind of depends on your maturity model like how mature your organization is with analytics and um how much statistical analysis you can do and whether your company even cares if you do that does that frankly so here's the thing to go from business outcomes to product outcomes you have to have a theory of how your product is going to create value for the business and so usually your business outcomes derive from your business model right so every business model has a set of metrics that measures the health of that business model And then what we're looking at is, is the product creating enough value to sustain that business model? And usually to uncover those product outcomes, we have to have a theory of how that product is creating that value. So for example, I'm gonna use Netflix in my examples. I've never worked with Netflix, they don't sponsor me. I just know people around the world are broadly familiar with them. Netflix has a subscription business model. So what are the metrics that are associated with the subscription business model? It's things like customer acquisition, lifetime value of a customer, where we're looking at things like retention and increasing retention, reducing churn, maybe average revenue per month. These are all variables that are business outcome metrics that support their business model. Now, the product team has to come up with a theory of how they're going to create value that will support that business model. And you can imagine for Netflix, that theory has something to do with engagement. But the more that you watch Netflix, the more likely you're going to keep that subscription. Now, as the product teams at Netflix grow and they break up that engagement metric into more and more granular metrics, do we need to evaluate, did this one team that was just focused on engagement with recommendations, drive retention. I don't know that that matters. Here's what does matter. Does driving engagement across all of your teams correlate with increased retention? So as you see customers engage, are those the customers that are more likely to retain? I don't know that we have to get all the way to the team level of like, hey, you clicked on a recommendation and that means you're more likely to retain. I think it's more about this whole theory of engagement will drive retention. We see evidence of that. And then we can see how each team is contributing to engagement. Yeah. Yeah. Yeah. Okay. Yeah. So indeed, right. Maybe not like being super accurate on like how much or even get to the point of, I don't know, comparing, because I've seen, for example, it may be different for each team has different product metrics, but then sometimes a business would also want to. quantify what that means in the business outcomes. So then you can also sort of, I don't know, rank the impact of all these different product teams. Right. And I think there is when it becomes quite difficult, at least not that simple. Right. Yeah. I see some companies try to like put a dollar value on each team. And I think that's really a result of our old IT model. We're thinking about teams as cost centers rather than as value drivers. and if we think about a product team as a value driver and the value and they're creating in this case is an engaged customer as long as we can show that engagement does in turn drive the business outcome that should be enough yeah yeah okay cool uh all right i will uh pick now the first uh question from the that we had on the page uh it's from lily baumgarten is uh lily on the call No? Okay, then maybe I will clear it out. So the question is, how can an opportunity solution tree compare opportunities that directly affect end users versus opportunities to improve internal development flow? And I think maybe it's related. There is another one, another question also on like, how do you even put opportunities related to technical debt, for example? Yeah, so this all depends on your outcome. Right? So the goal of an opportunity solution tree is to help you find the best path to your outcome. And sometimes technical debt is one of the best things you can do to drive your outcome. Sometimes it's not. We see the same thing with user experience, right? Any designer will tell you, You'll always benefit from improving the user experience, but that's not necessarily true, right? It's really will it impact the outcome you're focused on right now. So the idea of your outcome is this is the best way you can create value in the short run. And you want to look at the things that have the potential to impact that particular outcome. Sometimes that's improving the user experience. Sometimes it's launching new features. Sometimes it's paying down technical debt. But really the evaluation is what's the impact on our outcome? And I think, and then it's not just impact on outcome, right? It's also creating customer value. So the outcome usually represents business value. The opportunities represent customer value. So we're looking for what are the most important problems to solve for the customer? And sometimes we can't solve an opportunity. We can't solve the most important opportunity because we have technical debt. That's when we wanna pay down that technical debt, right? So what's nice about the opportunity solution tree is it's keeping us really focused on how do we create both customer value and business value. And then we're using that as our prioritization mechanism. And we will see bugs and tech debt and UX limitations come up in that context. And that's what tells us that, yes, we should work on those things. And the reason why I think that's so important is that we will always have bugs. We will always have user experience improvements we can make. We will always have tech debt. And we could spend our entire lives trying to pay that down and not necessarily create customer value or business value. Yeah. Yeah. And I think that's a good point, right? Maybe to also just put out there the technical depth that indeed will contribute anyways long term to that outcome all right i think we have another question of nina prijanka is nina on the call no but we got uh till uh in the call now so his next question is also very popular yeah do you want to make your second question the one for metrics Sorry, I'm actually calling in running between train stations, so I don't exactly remember what it was. Let me make it then. The question is, are there metrics for successful discovery, especially if you are in an industry that cannot roll out changes, experiments quickly? Yeah, so the best measure of discovery is are you making progress towards your outcome and how consistently are you doing that? The challenge with that... metric as a measure of discovery is that it's really noisy. You could be doing all the right things and still miss your outcome. You could be doing all the wrong things and still make your outcome, right? So when we would look at this as a metric of discovery, we want to look at the trend over time, not just how we did one quarter or another quarter. And it really is a relative metric. Teams that are better at discovery should be having a bigger impact on their outcome over time compared to teams that are not good at it. that are not that don't have a good discovery process. So this is a hard metric to really rely on, even though it is ultimately what we want to focus on. So there's this idea of when we work in an environment where luck plays a significant role, which is the case in business, we really want to focus on process. We can't just focus on the outcome. It doesn't mean the outcome doesn't matter, which is why I do want you evaluating. are you consistently making progress against your outcome? If you're not, there's things we need to change about our process. But there's also process variables we can measure. So the process variables that I like to look at are your cadence of discovery activities. So how frequently are you engaging with customers? Some people mistake this to mean how many customer interviews are you doing? But that's a vanity metric that we can game, right? So a team that does 12 interviews in the last week of the quarter is not equivalent to a team that does one interview every quarter. So I like to look at the cycle time between customer touch points. So can you reduce the days between the time you talk to customers? So we're increasing frequency. And then the other metric I like to look at is what's your turnaround time on an assumption test? Is it taking you three weeks to collect data on an assumption test? How do you reduce that time? And so I think both of those are good health metrics for evaluating your discovery process. But ultimately, if your cycle time on both is small, but you're wildly missing your outcome quarter over quarter, there's still something wrong. So we got to look at all three. Yeah, yeah. And I think you do that. So in the book, you also mentioned, I think that maybe the opportunity will, indeed, you will just find out that maybe it's not the right one to drive that outcome, right? Or maybe also try to even redefine the outcome, right? I think this problem space and solution space evolving together, I think you mentioned it on the book. Yeah, so this is part of the reason why outcomes are such a noisy metric, is that a team can be doing really good discovery. have a really rich understanding and opportunity space, do everything they should be doing to explore solutions and still run into a constraint that's outside of their control. Maybe that there's not technology available to fully address the opportunity they've been exploring and they discover that late in the process. Or maybe a compliance team comes and interjects and says, we can't, there's too much risk. liability risk here um and so those things will always happen or maybe you were doing great discovery work and then coveted hit and the whole landscape changed right so covet hopefully is just a black swan event we're not going to see more often in the future but we still see new competitors under the market that we can't predict we see new technology come about that we can't predict like we really do live in a very uncertain world and so there will be valid reasons for why we can't hit our outcome But what we can control is our process and do everything we can to be trying to make progress continuously. Yeah, yeah. Jim, I think you're playing the guitar to all of us. OK, yeah. Now you're muted. Some background music for everyone. All right. I think the next one is from Brian Salmon. Brian, are you on the call? No, then maybe let's take one. Yeah, I'm here. Oh, all right. I'm here, please. Hey, Brian. Hey, nice to be talking with you, Teresa. So I'm just a person that really learns by example. Do you happen to have an example of an opportunity tree map that's like kind of a full picture that we can see how all those leaves, how all those nodes on the tree come together? Yeah, so what's hard about this, this is probably the most common question I get, is that most real examples, companies see them as proprietary, right? It's their sort of expert knowledge about their market and about their customer. Most people don't want to share them. I'm starting to see that change a little bit. So there's some blog posts on product talk that definitely have some examples. There was a recent one. We did a product in practice article with Lisa Orr at Human API, where she shared a really early version of her opportunity solution tree. In the book, I tried to get into some more examples using a streaming entertainment company. um just to give a little bit more visibility into this um and then uh i think we have a post another product and practice series with super awesome where they shared some examples our most recent post um is uh hope gurion my partner shared some examples she sees working with teams so we're trying to get more examples out there because we know this is a real need um in january this past year i shared a really simple example from my own business But we are trying to get encourage people to share more examples, like maybe after the fact like this was the research you did and then you put out a product and now maybe it's a little more obvious to everybody that this was right so it's a little less proprietary. But because a lot of people think their discovery is a differentiating factor, they don't necessarily want to share that opportunity solution tree. All right. Yeah, I can see that happening. I think with research outcomes as well in general, right? It's very difficult that you see companies really sharing it or even OKR thing and stuff like that. I will share that I am working on a more detailed example, building out from streaming entertainment. So I actually started doing real interviews with Netflix users. Again, I don't work at Netflix. They don't sponsor me. It's just... everybody uses them and everybody's familiar with them and i am planning to share what comes out of that but it's going to take some time because it's definitely something i'm doing on the side but i do brian by far the number one question i get so we will be releasing more content around that in the coming months that would be awesome thank you yeah nice awesome all right i think next question is from magi peruch i think you are on the call Yes. Hi, everyone. And hi, Teresa. Thank you so much for writing such a great book with many various references. Thank you, Maggie. So I'm part of one of those big tech companies. And my question is related more to the overall strategy of a business unit and the team strategy. So how can we ensure that there is a connection between team strategy that is being set, what I call like at the top versus team specific OKRs? And then the second question is, should actually team strategy be set at the top rather than naturally derived based on short-term and long-term strategic projects? Yeah, so to both your questions, it really depends, right? So let's talk through this a little bit. So Marty Kagan talks a lot about the role of a product leader is to set the strategic context. He also says that most companies don't do this, right? So if you work in an organization where your product leadership is setting your strategic context, every individual product team needs to be constraining what they do based on that vision in that context. Right. So that's the ideal world. And you can imagine for a company that does this really well. you can imagine that like that becomes your guardrails. So you still have, you're still an empowered team. You're still tasked with an outcome. Your outcome probably connects directly to that strategic context, right? So one way that we do this is how we set outcomes. We set outcomes that are consistent with our strategic context. And then that sort of. set the boundaries in which you can explore and it's also going to have an impact on which opportunities you choose how you address those opportunities so it doesn't mean you're not empowered it just gives you boundaries to play within to make sure you stay aligned the real challenge here is that most companies either don't define that strategic context most leaders aren't defining that strategic context or it's not fully formed it's evolving and i don't think that's necessarily a bad thing because there's this There's this tension between a prescriptive strategy and an emergent strategy. And we actually want that tension, right? We want our leaders setting a vision, but we also want them staying open to what are our frontline teams learning and how should that be a feedback loop back into that vision. So the reason why I say this depends is if you are in a context where that strategic context is well-defined, what you want to do as a team is make sure that you're at outcome is aligned with that context and that as you learn, you're feeding back into that context. If you're more likely at a company where that strategic context just doesn't exist, you can actually start filling in the gap. You can start looking at, like based on my business model and based on the implied theory of how our product supports that, you can start making that implied theory explicit. and get feedback from your leaders like here's what i'm seeing across our organization this is what seems to be our implied strategic context and then you're giving them something to react to and that can help fill in those gaps and then same with usually most teams have some input in their outcome and if you don't i give you tools in the book for how to negotiate that right so that's another area where you can start to influence up on filling in the gap of that strategic context That's really good. Thank you, Teresa. I really like that reference to this emerging strategy. I think the problem with organizations is that they are either too vague or too specific that it makes us sort of like a squeeze. Yeah, so here's what happens, right? Most of the time our leaders think like, oh, my job is to communicate the strategy. And the way that I interpret that is I'm going to tell you what to build. It's very different from what I think Marty means and what good companies are doing, right? Is they're not telling you what to build. They're giving you guardrails within which to build so that what your team is doing is aligned with what the adjacent team is doing. And, you know, here's what I see. At the individual contributor level, our practice is really evolving. We're just starting to see that evolution at our leadership level, both with product leaders and with engineering leaders and with design leaders. And so it's just going to take time. And then if we think about the C-suite and how. other business leaders that aren't even in the product world, that evolution has barely even started. Right? So as individual, if we're an individual contributor, we have to acknowledge like that's our reality and start to fill in those gaps in the meantime. Thank you. Really, really valuable. Cool. Awesome. All right. I think next question is from Thomas Grundel. I think you're also on the call. Do you want to, yeah, please ask him. Yeah. I think we've kind of already covered the question about tech debt and infrastructure, but there's one underneath that that I'd love to hear your opinion on. The question is how distributed can a product trio be before it loses effectiveness? A tiny bit of context, I'm in an org where design is really one block of people, product is another block of people, delivery is another block of people, and we're not super integrated. So I might not talk to a designer unless I set an appointment really kind of. force it to happen at what point does that really start to harm an organization from day one is my short answer um i think the whole idea of a product tree is we want this group collaborating from the very beginning and if that's not the case you're really going to be limited by all the same limitations we saw in a waterfall world right if you're right if you have to schedule an appointment to talk to your designer That means there's not a lot of back and forth between a product manager and a designer, which means we have handoffs. Or there's not a lot of back and forth between an engineer and a designer, which means we have handoffs. So here's what I would say to you in that situation. You don't have to change your organization. Change your individual relationships, right? So don't worry about, hey, my organization is doing it wrong because you're going to bang your head against the wall and you may or may not have success there. I would look at who are my peers that should be in my trio and how do I start having more conversations and collaborating with them more. And the other thing I'll say is, so I was actually just on a call, I think yesterday or the day before where somebody asked me a similar question. They talked about co-location. So they basically said like, hey, I'm in one location, which is where I thought you were going with this. I'm in one location. My engineers are another location. My designers in another location. Okay, well, welcome to our world. For the last 18 months, that's been the case for almost every product trio. And what I love about this is that we used to believe we all had to sit next to each other to do good work. I don't believe that is true. I don't believe it was ever true. What I think we need to do to do good work is we need to have a lot of overlap in our working hours. Right? So we can't have somebody that's in a time zone 12 hours away unless we shift the hours in which we work. So we have to have enough hours that we can actually have real synchronous conversations. And we have to treat everybody as a first class citizen on the team. So we can't have co-located people have benefits over people who are remote. And we have to commit to using the now amazing collaborative tools that we have available to us today to make sure that we are culturally collaborating. So there's two parts to this. There's the relationship piece of even if my organization isn't setting me up to work on a trio, can I build those relationships? And then there's the second part of this is in today's modern world. And after the last 18 months of all of us learning how to work from home, we don't need to be physically co-located. We just have to be committed to working collaboratively. Thank you. The handoffs in particular at home. Yeah, the handoffs are brutal, right? The analogy I use for this is it's like a game of telephone. So like in our old waterfall world, a product manager is probably communicating with stakeholders. Maybe those stakeholders were talking to customers. So it's like customers to stakeholders, to product manager, to designer, to engineers. And by the time we get to the people building the product, we're eight hops away and the message is completely distorted. And are we really surprised we built the wrong thing? I don't know. It seems kind of crazy, right? Yeah, I think I agree, right? I think that was maybe the assumption test of the Trio, product Trio, having to sit in the same place. I think now probably not many believe that that's indeed still the case. Next question is from Michel Bautista. Michel, are you on the call? I think, no, but probably, yeah, I don't know, maybe you also get it quite often, right? How does continuous delivery impact roadmap communication? How can you keep the team well informed of what might come as you discover new things without the ability to sell or market anything just yet? I had also maybe a similar comment here, like, because I think that last chapter, I believe, of the book, right, you talk about showing your work and how you bring people into getting with you to the same conclusions rather than just telling them the solution. But I also was curious, like how you can do this, because I think from reading the book, it is on, you know, when you are on the meeting, right? Do this, but now maybe with more remote or work from home, how can we do this maybe in a more asynchronous fashion, right? How can we keep everyone in this, engaged with the discoveries? And maybe I think that the question a bit hinting at that, not getting everyone that excited that then sales start promising this to customers straight away, right? Yeah, so I'm going to start with the roadmap piece and I'm going to give a very ideal answer and then I'm going to give a more pragmatic answer. So the very ideal answer is if we truly adopt a continuous discovery process, we should not have a roadmap at all. Right. We don't know what we're building when. And frankly, if you started 2020 with a roadmap, I'm guessing by March you threw it away. Right. So. In order for that to work, though, in order for us to not have roadmaps at all, what's the purpose of a roadmap? Roadmaps help us coordinate across our organization. In particular, they help us know when to train our salespeople. They help our marketing teams set up campaigns and do launches. Here's the challenge. We're selling features instead of benefits. That's the problem on the sales side. And on the marketing side, we're... running campaigns around outputs and not around impact. So in an ideal world, our entire organization would make this shift to outcomes. Our sales team would sell solving opportunities, not specific solutions. And our marketing team would market customer success, not product launches. That's the ideal answer, right? Then you don't need a roadmap because the marketing team is running their marketing campaigns after launch, after they have successful customer stories to tell, and our sales team is selling customer success and addressing opportunities. Now, maybe one person on this call, if that many, works in an environment that works that way. So I'm going to give you the more pragmatic answer. The more pragmatic answer is we do still need roadmaps. We just need to have roadmaps that communicate the reality of the uncertainty that we work in, right? And so I personally really like the Now Next Future roadmap. Because what it allows me to do is in the now column, I can communicate. This is the opportunity we're working on right now. Because we're working on it right now, I can tell you what solutions we're building in delivery. And I might even be able to give you a date, depending on how much feasibility assumptions we've tested and how confident we are in our estimates and where we are in that delivery cycle. And my sales team and my marketing team can start doing campaigns around feature launches and my sales team can sell features because it's what's happening right now. And there's more certainty around it. In my next column, I might just say, this is the opportunity I plan to tackle next. Maybe we've started ideating and I can share here some potential solutions we might be releasing, but we're definitely not talking about dates because we're not in delivery. And we're definitely not talking about certainty around those solutions because we're probably still in discovery. And in my future column, I probably only have opportunities because we haven't even ideated or even prioritized those opportunities yet. So what that gives us is a little bit of this hybrid. We can communicate uncertainty, we can communicate where we are in discovery, and we can meet the short term needs of sales and marketing and everybody else in our organization. Yeah, yeah, indeed. I have seen that now on Next Roadmap. I also see a comment from Leo that is becoming, I think, more and more popular, right, to try to find this middle ground. Awesome. I will share even with a Now Next Future Roadmap. you will break promises. I shared a Now Next Feature Roadmap in January on Product Talk. And my Now column turned out pretty good. Like I put it in my Now column. It was all about the book and getting the book launched and my membership program and getting it launched. And that really did happen. My next column was about rolling out more courses around more of the habits in the book. And it turns out we're in late September and I haven't even started that next column. and it's because something unpredictable happened what happened is our existing course business blew up and we started to see um challenges with the way we were combining technology so i ended up with this like underlying infrastructure problem that if i started launching more courses on top of it i would have broken our world right and so i had to stop and say okay i need to fix these infrastructure problems before i add to them So that's one thing that I could not anticipate in January. And the other thing I didn't anticipate in January is that once you launch a book, people want international translations and they want an audible version. And that was nowhere on my roadmap, right? So even with a now next future roadmap, it's not going to reflect reality. It's going to change and it should change. right? It should change based on what you're learning in the world around you. And that's a really important thing to communicate with that format is the only thing we're communicating is committing to is what's in the now column. And everything after that is still uncertain. We're just giving you our best view of what we think will happen. Yeah. Yeah. And I think what I have heard sometimes is that when you deliver this roadmap, right? It's also a good opportunity to communicate what you mentioned at the beginning right to say this is anyways the outcome we're working towards right because maybe that also helps to start shifting a bit that mindset of uh i cannot give you deadlines i cannot tell you exactly where we're going to build and how it's going to look in one year right um awesome all right i think next question is from uh chris nicole you're on the call i am yes sorry please go ahead hi chris uh hi Thank you so much for taking the time to do this. It's really amazing. So I had two questions. I'm not sure which one is the next one, but I think I'll go with this one. In all the times that you've deployed with Teams continuous discovery habits, what are the most common struggles that you see teams go through as they adapt to these new habits? Yeah, this depends a lot on the organizational context. So I work with some companies. where really the first struggle is just getting access to customers. If your organizational context is this old world traditional waterfall, you're so removed from a customer that just finding a customer to talk to you is the first big hurdle. So that's one. Even though we say we're agile, we're not. We don't think in an agile way. We don't think iteratively. We still think in big projects. The way we get to agile is we take a big project and we break it down into two-week increments, but it's still a big project, right? So that's a big thing that I see is that in my coaching in particular, I'm really hammering on people to start to develop that mindset of what's the smallest thing I can start with. And this really starts with... really understanding the opportunity space and structuring the opportunity space in that tree structure, because the tree structure lets you break down big, hard opportunities into smaller and smaller opportunities, which helps unlock this continuous cadence. I would say those are the two big ones. There's probably a third one, which is related to the second one, which is when teams start assumption testing, they tend to think in these big project-based assumption tests. And there's a little bit of this like, learning curve of leap of faith that you really can learn a lot from a teeny tiny assumption test. And I think all three of those are just symptoms of the fact that most of business is still rooted in this project world and that digital product teams are like swimming upstream, trying to like break that mold and like force their organizations to be more continuous. And it just takes. until you see it or experience it, it's hard to imagine what good looks like. And that's, we started a series on product talk called product in practice, and we're trying to show what this looks like in practice for exactly that reason. Because we know that most people have never seen this firsthand. All right. You mind if I just ask a quick follow-up to that? Yeah. Because I think this is, you know, those are two very clear struggles. And I wonder, The second one of really breaking away from these big feature projects, what are some of the things that you start to talk to people about aside from looking at the opportunities and trying to find smaller opportunities? Like any specifics on sort of helping people break that mold? You know, it really does come down to deconstructing opportunities, right? You're not going to come up with a small solution if you're trying to solve a big problem. So you have to turn that big problem into a series of smaller and smaller problems. And one of the examples I gave in the book was about, let's say you're working at Netflix and you hear over and over again, I can't find something to watch. That's an evergreen opportunity, right? You could come up with, and odds are like your inclination as a product person is to come up with this bells and whistles, fancy solution to that problem. But as you interview and you learn about how do people already today evaluate what to watch? you start to hear smaller opportunities, right? Like I picked this show because I like this actor or I picked this show because my friend recommended it to me or I picked this show because the description was really compelling or because you told me it was character driven versus plot driven, right? And so the more you learn about how do people evaluate what to watch, you start to uncover these smaller and smaller opportunities like who's in this show or do I know anybody who's seen this show? Like these are smaller. problems to solve that lead to smaller solutions. You're not going to take a big opportunity like I can't figure out what to watch and come up with a small solution. That's just not possible. I think that breaking down the way until the small test exceptions, I think it's key, right? I think also to gain momentum probably, right? As you want to sort of like prove this new way of working to the company, right? I think that's probably also key. All right, next question is from Cor. Cor, I think you are also on the call. Do you want to ask your question? I'm driving actually, so maybe you could do this for me. Yes, I can do it. Sure. Yeah, thanks. All right. Yes, the question is, when prioritizing items in your opportunity solution tree, how do you deal with items that depend on each other? For example, we want to iterate and test assumptions for solutions quickly, but what if you need to... Due to dependencies, you need to develop multiple solutions before you can start testing. Yeah, so this comes back to opportunity mapping. You really want to frame your opportunities in a way that they don't have dependencies on each other. And this is not easy. I'm not trivializing this. This is where a lot of the work of opportunity mapping comes from. But your goal is to frame your opportunities in a way that you could pull a single opportunity and it does not have any dependencies on the others. So if I go back to that Netflix example, I can solve the opportunity of who's in this show without having to work on any other opportunity in that how do I decide what to watch branch. And that's what we're looking for. We're looking for opportunities. We're looking to frame our opportunities in a way that there aren't dependencies. And this is... This is a hard concept because most of us, again, work in context where we have like capacity planning meetings because our teams are structured such that everybody is dependent on everybody else, right? But this is a concept that's gaining popularity. If you've heard about the book Team Topologies, which is a really, becoming a really popular book, that book is all about value aligned teams. How do you create teams where they fully own what they're delivering? by themselves, no dependencies. This is the same idea behind microservices, popularized by Amazon, right? How do you create teams and products that own a piece where they plug and play with everybody else, reducing dependencies? Same idea with opportunities. How do we frame opportunities in a way they're not overlapping with adjacent opportunities? We really can pull one to work one at a time. I know this is probably not how your organization is set up, and it's gonna feel really foreign, It does take work, it does take practice, but the key is to frame your opportunity so that you don't have those dependencies. Yeah. I think it's also quite, I don't know if natural is a word, right? But I think on the book is also the example of that sometimes you will just have to take sort of like a solution at a time, right? Or like a small part at a time, right? So then you keep progressing and you see this final value only after. having taken or having delivered smaller chunks right so well i don't i don't want you to only see value after multiple iterations right a truly agile mindset is there's value after every iteration um again we're so steeped in a project world that's hard to imagine until you see it in practice right in a project world when we shift to agile we take our six month project and we break into two weeks sprints and every sprint does not deliver value. We got to wait all the way to the end of the six months. Some of us are better than that, right? We have a three week project and we wait three weeks before there's value. In an ideal world, every release is valuable in and of itself. The way that you get there is you work on smaller and smaller opportunities because even if the opportunity is teeny tiny, it's a customer need. If you fully address that need, it created value. it may not create huge value but it does create value and then you do that iteratively over time and that's what adds up to big value yeah yeah indeed um nice all right uh next question uh alice i think also with us on the call yes i'm here hi everyone hi theresa and i i was wondering if you could share some good practice for testing and validating opportunities in a b2b context um we are having some uh challenges uh due to not having statistical significance for a b testing and we we always go to the prototype testing but i was wondering if you have any other good practice to share with you us today Yeah, so the first thing I'm going to say is A-B testing is not a discovery tool. So don't rely on A-B testing to figure out what to build. Why is that? You just built everything before you learned anything. So A-B testing is not a discovery tool. I'm not saying you shouldn't A-B test. A-B test is a measurement tool. Once you've built something, it's going to help you measure, did this actually have the impact it intended to have? So A-B testing is a good thing. It's just not a discovery tool. In the book, I talk about assumption testing, right? We're testing our assumptions, and I give four ways to rapidly test assumptions. The first is you can prototype to simulate a specific moment. So if you're relying on prototyping, great. That's a great way to test assumptions. You don't have to worry about statistical significance when we're prototyping. Prototyping is a qualitative method. You're looking for signals. Are we on the right track? Second way you can test. assumptions is with one question surveys. If you embed these in your product, you usually can get a lot of data points really quickly. And it's just a fast way to test an assumption. Third way you can test an assumption is by data mining. Looking at existing data, are your customers already exhibiting the behavior they expect to see? And the fourth way is if you're testing feasibility assumptions, you can do research spikes around those very specific assumptions. In all four of these cases, it's very rare We're going to even in the survey case or the data mining case, it's very rare. We're going to be talking about large scale quantitative research. And the reason why that's OK is because we're not proving this is the right idea. We're comparing and contrasting. We're looking for a signal. And with each iteration, we're investing more time into our experimentation. And if you need a refresher on that in the book, in the testing assumptions chapter, I talk about this sort of. science piece like we're not scientists we're not running um randomized controlled double blind studies we don't have time for that and we don't need to do that because lives are at stake and we're also not creating new knowledge right we have really good feedback loops that tell us whether or not what we built worked So we do need rigor in our methods, but we do not have to always run large scale quantitative tests. And there's a lot in there. Like even in that chapter, I get into like the risk of small signals are false positives or false negatives. And I get into a little bit about how you can mitigate those. But don't be afraid of running small scale qualitative tests. That's a great way to learn. Great, thanks. Can I add a question or is it too much? I would like to hear more about your take on AB testing, when to apply it, in which context, just to learn a bit more. It isn't something I have worked with, but I always read a lot about it, you know, as the solution for problems. So I was wondering if you could share. Yeah, so again, it's going to depend on your infrastructure. So some companies A-B test every single release. So when they're releasing something new, they release it to a small percentage of their audience. Facebook's a great example of this. The reason why Facebook could have the adage move fast and break things is because when they ship code, they ship it to a teeny tiny percentage of their audience. They have an automated system that's evaluating. Does that audience perform in any meaningfully different way than the rest of the population? And they have a way to automatically roll back that release if they find a problem. If you have that infrastructure, you literally can A-B test everything. If you don't have that infrastructure, is it worth manually A-B testing everything? Probably not, right? So the key here, the key with the A-B test is to remember with the A-B test, we're measuring what's the impact of this release. And so if you have automated infrastructure, you can measure the impact of every release. If you don't, you're going to have to make some decisions. How much risk is there in this solution? How much of that risk did we mitigate in our discovery? How confident are we of what we learned in our discovery? Do we need to set up an A-B test before we release this? And that's a risk assessment. And if you're manually running A-B test, meaning your engineers are literally writing code that releases it to a percentage of your population, and you're using flags, and that's all manual at your company, you're not going to do that for every release. If you're relying on a third-party tool to do that, you might be limited in what that third-party tool supports. So you're not going to do that for every release. So it really will depend on your organizational context and sort of your maturity around A-B testing. I don't think A-B testing is the be-all, end-all tool. It's one tool in our toolbox that tells us what impact is this solution having. But we also have lots of other tools that can help us triangulate to get at the exact same thing. So don't think about it as if I'm not A-B testing, I'm doing something wrong. Think about it as this is one tool in the toolbox. Given what I have available to me at my company, what's the best way for me to mitigate risk? And if right now the best way for you to mitigate risk at your company is qualitative prototype testing, That's fine. Perfect. Thanks. And I think with B2B companies, especially what I found is that account managers or like customer success managers, this person that maybe have an account, they're always very helpful in scheduling interviews and having maybe help you get to this weekly cadence of research and interviews. Here's the thing too. A lot of us don't have the traffic to run reliable A-B tests. We just don't. Right? Like if you use... an A-B test duration calculator based on your current traffic, a lot of us, especially in a B2B context, your tests are gonna run for weeks. And here's the problem with that. No matter what, you're gonna be susceptible to false positives and false negatives. And the real way to A-B test to reduce some of that risk is to run your test multiple times. So now your three-week test is taking six weeks or nine weeks. This is not your most effective tool if you have a small population. And the reason why we talk about A-B testing so much is because the big companies have enough traffic to A-B test effectively. And that's we want to mimic all their methods. But if we don't have their traffic, it's not going to be the right method for us. Yeah. Nice. The next question is from Xenia. Do you want to ask your question? Yes. Hi, everyone. Hi, Teresa. I have a question around. if the team is like or is working with multiple personas uh how does it affect opportunity solution trees like do you have to create multiple opportunity solution trees or like what are your recommendations about it yeah great question we're actually in the middle of a blog series about this very topic so the first thing is one opportunity solution tree per outcome regardless of personas right so the opportunity solution tree is going to help you reach an outcome so that's the purpose so one tree per outcome Now, if you're working on an outcome, there might be multiple personas within that. My partner, Hope Gurion, just wrote a blog post on Product Talk that's live right now about how she integrates different segments into the tree. I actually have a different perspective from her, and I will be posting my perspective on October 6th. Is there a right answer? No. There's lots of ways to do this, and so we're going to provide some examples. Here's the key. You can have different branches on your tree for different segments. In some situations, that's the right thing to do. In other situations, if the journey across your segments is exactly the same, you might push the segments further down on your tree because the top level opportunities are exactly the same. Sometimes you learn about your segments and you realize one is more important than the others and you put that in your outcome. And now you don't have any segments on your tree, right? So again, it's really going to depend. But the root of your question, one tree per outcome, and then how different segments show up will depend on do they share a journey? Are their journeys completely separate? Is one more strategic than the others? And I recommend you check out that post by Hope because she provides three really specific examples. And then know on October 6th, I'm going to give my view on those exact same examples, and I'm going to introduce a fourth one. So that's a little foreshadowing. Thanks a lot. Nice. I see Leo shared the link, I think. Customer segments, Hope's take, right? Awesome. All right. Oh, someone asked, Mark asked, where are these posts? If you go to producttalk.org slash blog, Hope's post is our most recent blog. And then my post will be published on October 6th. Awesome. All right. The next question is from Jay Hawkins. I think you are also on that call. Yeah, hi. My question is kind of three questions rolled into one, really. From my experience, discovery un-earned information and data from a lot of different places, from anything from emails to sales cases to customer interviews, etc. The data tends to vary a lot. The challenge I've always had is knowing the best or the most valuable data to capture, where to capture it, and how to quantify that i've used a range of different tools and none of them have really really done it right so i was wondering if you have an experience or any recommendations on tools for that as well yeah so i'm going to tackle this in a couple of ways so first i'm going to talk about all the different inputs that we have how we synthesize that we're learning from them and then and then i'll get to tools last because i think the tools piece is the least irrelevant but i understand it's a it's a pain point okay so As product teams, we take in feature requests from our sales teams. Hopefully we're doing interviews. We have bug reports. We have support tickets. We have behavioral analytics. Here's my personal view. When we're mapping the opportunity space, I want to know the context in which those opportunities occurred. So if I'm just getting the data from my sales team or I'm just getting the data from a support ticket, I probably don't have the full context in which that need occurred. So I prefer to limit the opportunities on my tree to the ones I hear in interviews. Because if I'm, and there's a caveat here, I'm conducting my interview using the story-based. format that I talk about in my book, where in the interview, I'm collecting specific stories about the past so that when I hear an opportunity, I know the context in which it occurred. That's really important because I don't think we can address an opportunity without understanding the context in which it occurs. So I limit the opportunities that I consider to the ones I hear in my interviews. That doesn't mean our other data sources aren't valuable. I look at the other data sources as inspiration for what to explore in interviews. So I think about it as like what my sales team is hearing, what my behavioral analytics is suggesting, what my support tickets are suggesting. These are all like hypotheses of like these might be opportunities, but I want to make sure I hear them in an interview before I trust that they are opportunities. And that's twofold. If customers aren't talking about them and we talk to them in person, I don't believe it's that important to them. And two, it's giving us context. So that's the first thing is that I think about all those other sources as inspiration for what to explore in interviews. And then when I'm interviewing, how do I make sense of what I'm learning? In the book, I introduced this idea of an interview snapshot. So on a snapshot, you're capturing all of the opportunities you hear in a single interview. And then on your tree, you're capturing the opportunities from across your interviews that can drive your outcome. So we have two documents that are different. One is a single interview synthesis. That's your interview snapshot. The other is your cross the interview synthesis. That's your opportunity solution tree. So now let's talk about tools for each of those. I've seen teams. create snapshots in Google Slides where every slide is a snapshot. I've seen teams do it in Confluence. I've seen teams do it in Airtable. I've seen teams do it in Miro. The key is it needs to be a tool that everybody on your team is familiar with. I've seen teams do it in Figma. It needs to be a tool that everybody on your team is familiar with, that everybody knows how to use, and that is part of your weekly workflow so that they're visible and they're present. Same with the opportunity solution tree. A lot of teams use Miro or Mural. Some teams use Figma. Only use Figma if everybody on your team knows how to use Figma. So if your engineer isn't going to use Figma, don't use Figma, right? I've seen some teams do it in PowerPoint. I don't really recommend that. That seems really tedious and awful. But on the tools front, the key with both of these sort of synthesis points, everybody on the team needs to be well-versed in the tool, and they need to be present in your workflow. so that they stay visible and they stay living documents. And I wish I had like, I know so many people that like complain about Miro with opportunity solution trees because it's not really a flowchart tool and arrows don't move very well with boxes. Somebody's going to build a better tool eventually and we'll get better tools. But what I would say is just pick a tool so that your team can get well versed in it. And then magically when somebody down the line builds a better tool, we'll all switch to that. But right now, I think our tools are good enough. I don't think there's a perfect tool, but I do think there's many tools that are good enough. Great. Thank you. Indeed, I think that the key is maybe the communication or the access part for everyone in the team, right? I've also seen people just send it on the chat once a week, right? Just for everyone to see it, you know? Yep. Awesome. All right. I'm just conscious of the time. So for the ones that have the book, let's take a final photo group. So for the ones that have the book ready, if you can show it and then we will take a picture. I see Teresa, you have a lot of books behind. I love it. All right. So can I share a couple of resources? Yes, please. Yeah. So. Since it looks like a lot of you have the book and have either read the book or reading the book, I will share. So at producttalk.org slash blog, we blog twice a month all about one of our purposes right now is we're really trying to show what it looks like to put the habits into practice. So definitely check that out. We do also have a monthly newsletter. Both those resources are completely free. If you want more help than that, like we didn't get to your question, you want to connect with like-minded peers. Along with the book, we launched a membership program. It's $19 a month. That gets you access to our private Slack community. We do monthly community calls. They're kind of like mini coaching sessions. We do fireside chats with people who are putting the habits into practice. We geek out on tools. So that's available to you if you're interested in joining there. You can learn about that on producttalk.org. Look for Find Your People. And then we also have a variety of online courses that can help you build skill in each of the habits. So we have a course on how to... develop your interviewing skills we have a course on how to define outcomes we have a course on opportunity mapping so just know that as you're trying to put the book into practice you're not on your own we are really trying to help people uh adopt these practices awesome all right yes i think uh i i see people uh telling take my money so i can share all these links i think i have them all at hand um so i will share them on the on the email when i share the recordings as well And then, yeah, Teresa, if you have any other special link you want me to share, please just send it to me and I will send it on the group. Yeah, I added three links to chat. And everything is available from the ProductTalk.org homepage. So that's the only link you remember. But we do have a membership. We do have courses. We have a blog. Awesome. All right. Let's take now the picture with the book. All right. Everyone ready? Yes. Three. yeah three two one awesome thank you thanks everyone for joining thanks teresa of course very much for your time for all the helpful uh answers and yeah i will be sharing the recordings uh afterwards thanks everyone thanks for all the great questions

Some of the questions covered: 
6:30 How can an opportunity solution tree compare opportunities that directly affect end-users vs. opportunities to improve internal development flow?  
9:30 Are there metrics for successful discovery? Especially if you are in an industry that can not roll out changes, experiments quickly.  
13:50 Do you have an example of a FULL Opportunity Solution Tree that you could share and explain trade-offs of which paths to compare and pursue for development of JTBD ?  
16:40 How to ensure that there is a connection between team's strategy that is being set at the "top" and team specific OKRs? And should the team's strategy be set at the "top" or rather should it naturally derived based on short term and long term strategic projects?  
21:20 How distributed can a product trio be before losing effectiveness. i.e. Can representatives from departments come together intermittently on a project basis and achieve the same success? Is this likely to work?  
* How do you prioritize and represent major tech debt and infrastructure initiatives using an OST?  
* What are the metrics for measuring product success and customer satisfaction - how useful they are?  
25:20 How is continuous delivery impact roadmap communication? How can you keep the team well informed of what might come as you discover new things without the ability to sell or market anything just yet?  
* On showing your work to stakeholders. Any tips on how to do this continuously on an async manner? (Not on meetings)  * What type of customer research should be conducted and how often?  
* How should we decide what and what not to build?  
32:00 In all the times that you've deployed Continuous Discovery Habits, what are the most common struggles that you see teams go through as they adapt to these habits?  
36:50 When prioritizing items in your Opportunity solution tree, how do you deal with items that depend on each other? For example, we want to iterate and test assumptions for solutions quickly, but what if you need to due to dependencies you need to develop multiple solutions before you can start testing?  
40:30 Teresa, can you share good practices for testing and validating opportunities in a B2B context? How to have statistical significance when A/B testing would involve a smaller number of users? 
47:45 In case you work with multiple personas, do you recommend creating multiple solution trees?  
50:10 discovery unearths insights from everywhere. My biggest challenge is knowing what information to capture, where to capture it and how to quantify it. Do you have any recommended tools to help with this?  
57:55 Picture time! :D  

All questions submitted here: https://productbookclub.com/book/continuous-discovery-habits  

About the Product Book Club:  
Discuss product-related books once a month with new product colleagues and the book's author. 
Join productbookclub.com to know what other books we are reading, and participate in the upcoming discussions!  

If you find our discussions or content useful, please consider becoming a patron: https://www.patreon.com/productbookclub  

Follow us on Social Media: 
https://www.linkedin.com/company/productbookclub/ 
https://www.facebook.com/productbookclub 
https://instagram.com/productbookclub 
https://twitter.com/prodbookclub