Customer Interview Analysis - All Things Product with Teresa & Petra

Hi folks, this is All Things Product with Petra Wille and Teresa Kortz. And we're so happy you're here. Teresa, in one of the recordings this spring, I think we both hated the idea of analyzing customer and user interviews with the help of AI. Did your opinion maybe change on that topic ever since? It's half a year later, so... Yeah, it's always evolving. Let's start with the experiments I ran, I think, last April. I had just conducted 15 interviews with product leaders, and I was doing the synthesis myself, and I was really curious how good is AI at helping with this. And so I did the synthesis on all 15 interviews myself, and then I ran them through ChatGPT and through Claude. and I compared both results to my results. So I did it as like an experiment. And I tried to like with both tools, I tried to like set up projects and give it context. So like it would try, I tried to teach it to synthesize an interview like I would synthesize the interview. And ChatGPT at that time, this was April of 2025, was terrible. absolutely terrible. It made things up, it fabricated quotes, like it hallucinated all over the place. It was terrible. Claude... I remember we were both really against it back then. Yeah, Claude was surprisingly good. So Claude could wrap its head around, its metaphorical head, around opportunities and what opportunities are. So for people familiar with my discovery work, I basically was trying to teach an LLM how to create an interview snapshot. So identify opportunities, key moments in the story to generate an experience map, maybe a memorable quote, some quick facts to like do segmenting and understand where the story, the context in which the story occurred. So a couple of things, working with story-based interviews and then really trying to synthesize, grab all those interview snapshot elements. And Claude was interesting because with some effort, So with giving it the right context, I could get it to be like, let's say 60 to 80% good. And it found some opportunities that I missed. And I did go back to the transcripts and verify they were real opportunities that I actually missed. I found some that it missed. So that's, I don't want to just rely on Claude, right? Because I found things it missed. So joint venture is still the thing? Yeah, sometimes it found things. Yeah, sometimes it found things that it framed him. Like we found the same things, but Claude framed him a little differently. And that was interesting because it forced me to think about Claude's framing, my framing and which one's better and which one feels more accurate given the transcript. And so my takeaway at that time was, okay, Claude actually is a really good teammate. Like I don't want to just... thinking partner right as we were saying in one of the last episodes as well a thinking partner yeah so i was like okay i don't want to outsource this because i certainly found things claude didn't but claude is a super helpful partner and so i think i wrote and we probably even did an episode about like think about claude as another person on your team everybody synthesize the interview individually and then um we'll go from there yeah my thinking changed a little bit and this is because of a couple of studies. Okay. Before I get into the studies, let me just talk about language because I keep saying synthesis and this episode is called analysis. So I'm going to nerd out on words for a minute. Bear with me. We titled this episode customer interviews analysis because our keyword search tells us people search for that more often than customer interview synthesis. Which I would say has to do with non-native speakers. So many of us are non-native speakers and we're using analysis more. Yeah, so let's talk about analysis and synthesis. So analysis is usually like what's in this interview? Like what happened in this interview? Like it's just factual what happened in this interview? And synthesis is more like what does it mean? Right? So I would argue we are actually doing both analysis and synthesis. My bias is I tend to put more emphasis on that second step. You can't really do synthesis unless you've done analysis. And you definitely don't want to only do analysis. You want to do both. So that's my little word nerd sidebar. Yeah. Thank you. Thank you. Linguistics, important. Yeah. Okay. So let's talk about some studies that have come out. So there's been a number of studies. I'll try to track these down and put them in the show notes. Where... They look at like people who use AI at work versus people who don't use AI at work. And these are like controlled studies where they're actually giving a group AI tools and telling them how to use them in their work and another group where they're not. And what they're finding, and this is turning out to be a fairly consistent finding, is that AI helps raise the floor of our lowest performing employees. If you have no skill, if you're junior in your career, if you have less experience, if you're less skilled in an area, AI raises the floor, right? Which does make sense, even with your findings from spring. Yeah. Yeah. These same studies are finding, however, that our experts, when they also use AI in expert ways, and that's the important part, they get even more benefit. Maybe not even more benefit because the bump from junior to competent might be a bigger jump than expert to expert plus AI. But the expert plus AI is outperforming everybody else. So the highest performing combination is somebody with expertise plus using AI competently. But I would, so from, yeah, and from personal, just real quick personal story. I was, this is now really nerdy story to all the listeners out there, but I today run into the problem that my stream cam is no longer working with Riverside properly. So I decided to upgrade my camera. And then I learned that the USB ports that I'm currently having at my computer here at home are not good enough to actually. have such a camera connected. So I needed to upgrade my notebook anyways, which is old. It's my second setup here. We always record in the evening. So I'm at home. That's not my office setup. So I was like, maybe I should get a new notebook. And then there was too expensive and chat GPT and I were riffing about maybe a games PC because I had some certain requirements and maybe we can put the configuration down. And it was super easy for me because I know the language of how to put together a new desktop PC. So that's what I've done all my all my life, basically. So I have a lot of expert knowledge on how to do it. But I was not fully sure about the latest trends and what is the things that I really need and where would I overpay for the things that I have. So it was easy for me to use chat to configure that. And then on the other hand, recently, I did some home planning. and I have certain skills in that field but nothing really expert level I would say And it was nearly impossible to get something decent out of ChatGPT or Claude. So even not with Claude Code where I experimented with that a bit as well. So it didn't took me somewhere. And I think it's just because I'm not an expert in the field. I don't know how to prompt properly in that area. I don't know how to provide proper context in that area. And it was so much easier with the PC configuration game today where it was really helpful and it was really quick and easy to set up. Yeah, let's actually use this PC example because I think this is a great way to illustrate some of this research. So let's say you know nothing about computers and you're about to go to a computer store and you want to learn a little bit so you don't get snowed by a salesperson. You could as a totally beginner, ask ChatGPT, like, hey, I need a new computer. Here's what I'm going to use it for. What do you think is a good baseline? What should I be looking at? What should I expect to pay? And ChatGPT is going to raise your competence from a beginner. to like a novice, like a lower intermediate. So that when you go in that store, you get a better outcome than somebody who's a total beginner. But we are seeing in multiple studies now that like it's pretty darn good at beginner to like lower intermediate. The challenge is like, did ChatGPT give you a great response? You don't know because you're not an expert. You can't actually evaluate how good the response is. What's funny is there's this middle ground of like, okay, maybe I have a little bit of knowledge and I can see that like what ChatGPT gave me is okay, but it's not great. But I'm not an expert, so I don't know how to push it from okay to great. Get better results. Yeah, exactly. Whereas if I'm an expert in building computers, I can look at the result and be like, okay, I like 90% of this, but I'm going to fix the last 10% or I'm going to ask ChatGPT to change the last 10%. And so that's where you can see like if you're an expert plus AI, you get even more because like maybe the AI thought of something you that's like a blind spot for you even as yeah it did right yeah it did but it still suggested the most ugly cases which I briefed him not to do and it still did so I had to google for that myself in the end but yeah the rest was fine so now we can get back to customers interview now please sorry for the detour folks here's the problem I see. A lot of teams when they're new to customer interviews, they follow my advice, they interview a customer every week, and after like four or five weeks, they're like, we have so much data. We didn't do any analysis or synthesis. It's building up. We don't have time. We want to stop and catch up on what we've learned. And there's a couple problems with this. You have to do your analysis and your synthesis as you go. Like you have to do that. If you stop to do it, you're never going to start again. And I can tell you this because I've seen it hundreds of times. Right. So there's this important corollary to the interview a customer every week habit. It's do your analysis and synthesis every week along with it. And I understand why teams don't do it. Analysis and synthesis. I'm going to start just saying synthesis because saying both is hard. Synthesis is like synthesis is hard and it's time consuming. And to do it well can be really time consuming. And like teams just don't have time, right? Yeah, and especially doing it as a team. Yeah, exactly. Yeah, so that is the time consuming part. What's happening in practice, like in my Teresa's ideal world, a product team would interview every week, they would synthesize that interview every week, they would do it again, they would do it again. Every three or four interviews, they would update their opportunity solution tree, and everything would be great. Patrice's ideal world does not exist. Right? Damn it. Yeah. Teams don't do the work. They get stuck. They run out of time. Interviews pile up. So this raises the question of like, okay, well, should they use AI for this? And man, I have so many mixed feelings about this, but they're changing. And so I wrote a blog post about this, about the first challenge I see with AI customer interview synthesis. Is your interview any good? And like, here's the hard reality. Most teams interviews are not that great. And like having AI synthesize it doesn't fix your bad interview. So like the first thing we have to look at is like, are we conducting competent interviews? And nobody wants to hear they're not very good at interviews, but they're not very good at interviewing. So then the second thing is- Wait, wait, wait, Teresa. So would you suggest people using AI already to assess their research slash interview questions? Yeah, I mean, I have an interview coach that can use for that. I know. Glad you brought it up. I do have an interview coach if people want to get better at interviewing. That was not the goal of that whole rant. But I do, and we'll put it in the show notes. Once you get competent at... interviewing now like maybe ai can help but what a lot of people do is they say okay i'm going to dump all my interview transcripts in a like notebook lm or in a clod project and i'm just going to ask questions of my research okay that's cool i get it but like that's not very structured synthesis like one of the things i teach is you need to you need to synthesize each interview separately. Yeah. Like what did I learn from this customer? What did I learn in this story? And then we need to synthesize across our interviews. What are we hearing across these stories? Yeah. And like, yeah, because again, context matters for every interview. Sorry, delays here. Absolutely. Yeah. So if we dump all this into a folder and we just ask LLM's lazy questions. we're really not getting the most value out of our interviews. Yeah, I agree. And so like, can AI help? I think AI can help with some caveats. Like we have to set up the right environment to teach an LLM how to synthesize an interview across both these steps. First, help me understand what I heard and what I learned from this single interview. From this one person. I can do that for each of my interviews. Then I can say, okay, now help me understand what I'm learning across my interviews. And then, okay, so let's say I don't have any synthesis skill and I'm a total beginner. I think we are learning that AI is probably better than you doing nothing. And I'm so reluctant to say that out loud because people are going to walk away and say, Teresa says I can use AI to do my synthesis. It's fine. Teresa said it's fine. I don't want to do it myself anymore. Right? But like, if you don't know how to do any of this or you're just outright skipping it. It's probably better to use AI than to use nothing. With the caveat, you have to teach the AI how to do it well. Yeah, and I could maybe add, I would like to add one dimension, which is I would rather prefer a team to use AI to synthesize interview findings, but then invest time in creating a story. for the product to basically bring the rest of the team along and the rest of the company along. And that needs to be a really tangible human infused story. And if what they synthesized out of the interviews, the quotes, the findings can become this really human stories of where people are struggling and where people have real problems that the company could be solving. I think. That, for example, would be a time shift that I would support to say like a bit less synthesizing a bit more help from AI over there. But then with the storytelling part, and again, you can use AI as a thinking partner, even for your storytelling part. But this is where there is a lot of art and you're usually missing the human touch if the team is not actually doing that well. You're already laughing, Teresa. Yeah, because I think this is where our biases come out because I would rather you do the human synthesis, the interviews, and let AI help with the storytelling. I love it. But I love it when this happens in our podcast that we both uncover our own human biases in our work and our experience. That's why we're doing it personally. Here's where I think it gets really interesting. I do think it's true that... AI synthesis is probably better than total beginner synthesis and definitely better than no synthesis. Yeah. But here's the crux of it. If I personally believe how well we understand our customers is our best competitive moat. I think the companies that do a really good job of understanding their customers and their needs, that's one of the best competitive advantages we can create. So do we want to just be competent? at this? Like, yeah, tick the box off we've done it. Tick the box off that we've done user interviewing. That's another reason why people want to automate it so much. I'm really interested in this like expert plus AI synthesis. So can we create a workflow where we help people develop synthesis skills? so they're not beginners anymore. They become competent experts and we pair them with AI and AI helps us make it go faster, but we're still, we get that highest level of performance of AI expert plus AI. And that to me is like the most exciting thing. And so in that one post- Yeah. I have one question, which is we both still are against the AI interviewing the users, right? Just to make that clear. Who's conducting the interview? Put a pin in that for a second, because that's complicated. We can come back to that for a second. But one of the things I did in my recent blog post, which we will put in the show notes, is I tried to paint a picture, a very clear picture of what the future might look like when we have competent, skilled team product team members that know how to synthesize an interview where the AI is facilitating and contributing to the synthesis so that the team can do fast synthesis. And so I painted this picture of like, let's say you finish a 20 minute interview, you spend 15 minutes more as a team with your AI buddy synthesizing the interview and you can do it all in 15 minutes and you're getting expert output. because you're using your expert skills and you're using your AI skill to help you. And so what I would argue is that like, yes, if you're a beginner and you're doing nothing, yes, AI can help you if you set it up well. But I also want to encourage people to like invest in their synthesis skill because this pinnacle, like the peak performance is going to come from you being an expert paired with AI. Yeah, and this is where the innovation usually comes from. The innovation usually comes from this aha moment where an engineer sees engineer person in the team sees somebody struggle and having this epiphany of we can easily solve that. And this is so often where a competitive advantage comes from or something like this is, oh, my God, this is so easy to solve. If only we would do this or something like that moments. And I think. these ones are easier to spot for the people holding all the context in them already because working for the team on the product for quite some time. So you still need this human lens to some extent. Yeah. Okay. So now we can come back to your question of who should conduct the interview. Yeah. In April, I would have absolutely said that someone on the team needs to conduct an interview. And I'm still 90% there. The reason why I'm open to the 10% is for the same argument we just went into with customer interview synthesis. If you're on a team and you're not interviewing customers and you're not going to interview customers, is AI interviewing customers for you better than nothing? Probably. Is that going to give you very much value? I don't think so. I think one of the primary benefits of interviewing customers is the empathy you get from direct contact with your customers. which you're not going to get from AI. No, I agree. But if you're literally doing nothing and you're unwilling to do anything, could you get some value out of these AI tools? Probably. Here's the thing though. If you're on a product team and you're building a product that's going to be used by humans and you're really trying to find any path whatsoever to not have to talk to your customers, I really want to encourage you to sit down and journal about this. Why? Why are you like, isn't I always think maybe this is just my naive ideal self, but like I always think that like as product people were excited about serving our customer. And so like, why are we avoiding them? Don't we need to learn about them and like engage with them and have empathy for them and like understand who they are? And like, if that's not appealing to us. Yeah, says the most introvert person, right, Teresa? So even if you are a really introvert person, you're like, okay, but my scientific brain wants to learn from the humans out there. And just by observing them and by observing some of their struggles. So why are we in these jobs, right? Like, why are we building products for people if we're not interested in who those people are? I agree. This is a philosophical discussion now. This episode is growing too long. I know. So the 10% is because I can see how AI interviewers could play a role in our toolbox. I do not think it should ever replace the product team actually talking to customers directly. Yeah, I would so much agree. That was really insightful. Thank you. And by the way, thanks for changing your opinion. And thanks for sharing with the world that you are changing your opinion to some extent. It's evolving. I wouldn't say it's totally okay evolving. Okay. Let's put it like that. Linguistics matter, you know. Thank you, Teresa. Thanks, Petra.

In this episode, Petra and Teresa revisit a topic they once strongly pushed back on: using AI to analyze (and maybe even synthesize) customer interviews. Six months and several experiments later, Teresa’s perspective has evolved — and the conversation opens up a nuanced look at where AI can genuinely help and where it can quietly erode your team’s customer understanding.

They dig into recent studies on how AI affects junior vs. expert performance at work, the difference between analysis and synthesis, and why your unpolished interview skills matter more than any shiny new AI workflow. Along the way, they share personal stories (including an unexpected gaming-PC detour) that illustrate how expertise shapes the usefulness of AI tools.

This is a candid, practical conversation for product teams wrestling with the real-world tradeoffs of integrating AI into continuous discovery.

Show Notes
In This Episode
🔘Why their early stance on AI-powered interview synthesis has shifted
🔘What Teresa learned from running 15 interviews through ChatGPT and Claude
🔘How AI raises the floor for beginners but accelerates experts even more
🔘The importance of separating analysis from synthesis
🔘Why most teams struggle with customer interview synthesis in practice
🔘What happens when interviews pile up — and whether AI can realistically help
🔘The risks of relying on AI when your interviewing skills aren’t solid yet
🔘A vision for “expert + AI” synthesis that’s both fast and high-quality
🔘The ongoing debate about AI-led customer interviews
🔘A detour into PC-building that perfectly illustrates the limits of AI support

Key Takeaways
🔘AI isn’t magic. It can help, but only if your interviews are strong and you provide the right context.
🔘Beginner + AI is usually better than nothing. But the real performance gains come from expert + AI.
🔘You still need to synthesize every interview individually. Dumping transcripts into an LLM isn’t a shortcut.
🔘Customer understanding is a competitive moat. Outsourcing it entirely will cost you in the long run.
🔘Empathy comes from human interaction. AI can’t replace the experience of talking directly to your customers.

Resources & Links:
Follow Teresa Torres: https://ProductTalk.org 
Follow Petra Wille: https://Petra-Wille.com

Mentioned in this episode:
ChatGPT: https://chatgpt.com/
Claude: https://claude.ai/
The Interview Snapshot: How to Synthesize and Share What You Learned from a Single Customer Interview: https://www.producttalk.org/interview-snapshot/?srsltid=AfmBOoora_R1Pqm7IDMagCZuhQBJAZQ_qGU22wo7BxKK00RSR0JhxwNQ
The Interview Coach by Teresa: https://learn.producttalk.org/course/story-based-customer-interviews
Customer Interview Analysis: Where AI Helps and Hurts by Teresa: https://www.producttalk.org/customer-interview-analysis-ai/

Have thoughts on this episode? Leave a comment below.
#AllThingsProduct