AI Engineering - All Things Product with Teresa & Petra

Hi folks, this is All Things Product with Petra Wille and Teresa Chorts. And we're so happy you're here. Teresa, in one of our last episodes, we discussed that nowadays everybody could become a product builder and code more and work more on the actual product, but maybe not everybody wants to. But rumors that I heard say that you are a full-time engineer now. So maybe you can share a bit more about that journey with us. Yeah, it kind of happened accidentally slowly over the past year. But I would say I spend 90% of my time. Well, I still write a lot. So maybe like 60-40 is probably more accurate. I probably spend... 60 percent of my time literally doing engineering work and probably 40 percent of my time writing and engineering work is that in terms of really working on a product yeah so it started with the interview coach i've written and shared a ton about me just building my first ai product and learning and um it just grew from there i i'm trying to think of like how it happened i built my first ai product starting in march of 2025 and then i kind of just kept tinkering and playing but nothing that was really real like i started playing with my own personal productivity workflows that's why i started nerding out on cloud code but then i started talking to vistally and vistally makes opportunity solution tree software their founders have been part of my community for a long time so i've gotten to see like how they grow have they've grown and gotten to know them really well And they were like, Teresa, you've been doing all this experimenting. Maybe we could integrate the interview coach individually. And so we did that. That was like a huge step up because I was like, oh, this little toy that I built for my course is now going to be in a production product. I better learn like real engineering skills like error handling. But you and I know that you already had them from your past. No, I did not. I actually did not. I had been a front-end engineer. in the past but that's very different than like rock solid like engineering in a production environment make it always work and so i literally had to learn like how do you do good error handling how do you do automated testing like when i was a front-end engineer nobody was doing automated testing that wasn't even a thing yet I didn't know anything about a CICD process. I still barely know something about a CICD process, but I'm learning. I had to learn about security. I had to learn about efficiency. Claude's great at writing code, but it's not very great at writing code that's going to perform at scale. That's an area where I've had to really question it and be like, what if we had 40,000 records? What if we had 400,000 records? What if we had 4 million records? And usually it will design for the like 4,000 records. And I'll be like, okay, well that doesn't cut it. Anyway, so it started with the interview coach. Like I talked to this company, I created this partnership and then I immediately like scared myself. I was like, am I even capable of doing this? And I learned a lot and it turned out the interview coach was integrated. It was fine. And then we started talking about, I started running experiments about could I do AI generated interview snapshots? Could I do AI generated opportunity solution trees? And that was like for this delete. And then in the meantime, for my courses, I started to experiment with like, well, what else could I add to my courses? And I built a business fundamentals coach for our new course, Business Fundamentals. I recently built an outcome coach. So you tell it your outcome and it gives you feedback on your outcome. And I just realized like I am now like most of the time engineer. And I didn't. What will you be doing? The question is, are you running out of ideas and stuff to put into code real quick? Or is there a long backlog of things that you still would love to work on? I'm not running out of ideas. So I'm building on two paths. So there's the Vista.ly path. So I now have a partnership with Vista.ly. They basically license AI services from me. Right now, those services are interview snapshot generation and opportunity solution tree generation. but we're also working on an AI interviewer. We're working on, I can't share the longer term plans, but there's some like right behind that is a really big idea that I'm very excited about that I'll be able to share eventually. And that's like the Vistily path. And what I love about the Vistily path is Vistily, everybody keeps asking me like, why don't you release skills to help people do this? Well, I love Cloud Code and I love the personal productivity part of Cloud Code, but Cloud Code isn't multiplayer. So I believe discovery is a team sport and you need a tool that allows everybody on your team to add their interview notes and to add the thoughts to the snapshot and to see the snapshot and to add comments to the tree and to contribute to the tree. And so like it has to be in a collaborative interface and that's what Vistaly builds. So the reason why I have my partnership with Vistaly is like, I don't want to build all that stuff. I don't really want to be a software company. And so our relationship, I'm almost set up like an AI researcher. I figure out what the AI can do. You're the innovation lab to some extent. Say that again? You're the innovation lab. Exactly. And then I license it to them. And that wasn't really great. Because I don't have to worry about SOC 2 compliance. They already have that. I don't have to worry about like data compliance with GDPR and the EU. They already have that. I just get to like... play with what's possible build the ai service around it and then they deploy in their environment that's got all the compliance going on so that works really well then i have this second um thread going where i have this really grand vision i so historically i've run a training company over the last i talked in a previous episode about how i cut uh five of our seven programs you were sun setting them and basically i'm trying to cannibalize myself So I think the future of training is you'll have an AI agent that will give you just-in-time training as you need it on the job. And so I'm building basically a Teresa bot that has all of the skills it needs to be your discovery coach. But one that is not only based on a 40-minute podcast episode. No, not one that says ChatGPT act like Teresa. One that has access to literally everything I've ever written. everything I've ever recorded. That's its foundation, but that's not enough. That's great. That's like going to Google and searching, what has Teresa said about this? The reason why I built an outcome coach and a business fundamentals coach and an interview coach, and eventually there'll be an assumption testing coach and an opportunity mapping coach, right? The reason why I'm building all of these, these are going to be tools for TeresaBot. So that when you go to TeresaBot and you say, blah, blah, blah, I'm working on this thing. I have this hard problem. it knows which coach to invoke to then coach you. I love it. And so this is a whole different, so I have like two paths and I'm working on them in parallel. But it's all engineering. Like it's all really, it's all AI engineering. It's all really learning context engineering, prompt writing, orchestration, evals, observability. Amanda, do you find good enough resources for you to be easy to learn all these new skills? Well, I started a podcast called Just Now Possible. I know, I know. And it's like my cheat code. Because I hear a team talk about how they solved a hard problem. And in my brain, I'm like, oh, I can apply that to my work this way. So that's the first thing is I created like the best cheat code if I found a way to interview some really good teams about how they're building AI. Yeah. What's fun about AI engineering right now is that it's like the 90s with the web. Everybody's learning it together. There's a lot of people blogging. There's a lot of YouTube videos. There's a lot of podcasts. Slop and a lot of slop. There is. So you got to pick your sources for sure. But honestly, the way that I'm learning is by trying to do it. And then I'm letting Claude teach me. So I'll give an example. I built the foundation for Teresa Bot maybe like... two months ago. And I had never done anything with RAG, which is Retrieval Augmented Generation. I'd never done anything with an embeddings database. I had no idea what's the right way to give my agent access to all of my content. I knew those things existed. I've been talking to teams about those things. Through Just Now Possible, I've learned what's hard about embeddings, how do you have to do it well. So I wasn't a total beginner. but I had never done it myself. And so what I did was I just started a conversation with Claude and I actually published this conversation on Product Talk. So if you go to producttalk.org, I have an article about vibe coding best practices. And this example of how I built Teresa Bot, the full transcript of my back and forth with Claude is in that blog post. Oh my God. And I literally just started with like, here's an idea that I have. Help me think about how I can make this real. And we just went back and forth and designed an architecture for it, designed like the first mini experiment, which is let's just put 20 blog posts in it and see how good the search is. And I built the first version of Teresa Bot, which is basically just an embeddings database with a rag step in like two days. And I launched it inside my Slack community and got real people testing it within like 48 hours. just quickly folks can you hear how teresa is still following best practice approaches and still having conversations with real users we are not skipping that step not even when we're developing ai products i just wanted to get that one out yeah it's it's actually um you know last summer i took hamel and shreya's class ai emails and um I was barely like, I was like a total novice on AI products. But you know what I love about that class? I actually think Hamill and Shreya think about AI products the same way I think about discovery. They've done a really good job of identifying what are the basic skills or basic habits that are universal and then how you apply them changes a lot of different ways. And so I feel like I got... a really good foundation. The other thing I learned from that class, and this is because I have gotten to know Hamill and we've had a lot of side camp conversations. I think that I am a data scientist at heart and just didn't know it. So like if I think about discovery. What makes your discovery good is diving deep in the messy qualitative data and diving deep in the quantitative data and figuring out where's the signal in this noise. And if you think about AI products, people that just write prompts and release something don't realize this yet. But if you want your AI product to be good, you have to log traces. You have to look at your data. That's Hamill's mantra. And you have to really dig in and not just like... tweak your prompt a hundred times, but dig in and understand what's actually going wrong here. So I'll give an example. Yeah. And then trust your brain with the eureka moments. So you need... Yeah, exactly. And so like what Teresa bought, I, to like test it even before I put it in front of users, I generated like a hundred questions I thought users would ask and they actually came from my Slack community. So they were a hundred real questions people have asked me in the past. And then I ran Teresa against those questions. And then I looked at what were the errors it made. And I systematically identified the errors and then used that to iterate on the prompt to change the orchestration, to change the context, to improve it before it even saw users. And this process to me is so analogous to good synthesis and discovery. yeah i'm pretty much a most of the time ai engineer right now but i don't feel like the work i'm doing is different from the work i've done in the past in the past i was working with interview transcripts and um behavioral analytics and support tickets and sales conversations and now i'm working with traces like ai traces but i feel like the process is exactly the same and so i think like maybe what i didn't know is that maybe product managers if you want to be good at discovery we need to like build some basic data science skills. And so that's been kind of fun to like uncover like, oh, maybe I always was the data scientist. Yeah. Yeah. I can only second the moment that I learned more about data science, all of my discovery work became so different, way easier. And innovation was really happening because I had I was better equipped to surround myself with all the data in. forms that really helped me to see whatever we need to build for the users. So that actually was a data scientist who made all the difference here. So I agree more of that. I want to come back to something you said earlier, and this is something I'm seeing more on the internet. So you said, but you've been an engineer before. And so I want to talk about this because I think some people are using this as like, yeah, but Teresa's different. I can't do that. I want to talk about how I was an engineer before. So I have an undergraduate degree from Stanford. Stanford has a like best-in-class computer science program. I did take computer science classes at Stanford. That alone makes you different. But here's what I took. I took two beginner classes. Right? I didn't take four years of computer science. I took two beginning coding classes. and the rest of the computer science classes i took were all the math theory side of computer science they were not programming related interesting so i didn't build engineering skills because i wasn't a computer science major i was a symbolic systems major so the goal of the classes that i had to take were to introduce me to this perspective on the same ideas the rest of the program covered so like i didn't come out as a college student an expert in engineering what i came out of college as was a designer because my focus was on human computer interaction so i had a little bit of technical skills but i mostly came out as a designer the problem is i graduated in 1999 there were very few companies hiring designers what they were hiring was front-end engineers and i had learned on my own time not at stanford on my own time because the internet was a new thing i had learned html css was starting to become a thing i was learning css The first company that I worked at had its own proprietary template language, which by the way, the first three companies I worked at had this because this was before front-end frameworks, like we had to roll our own. And so when I say I was an engineer, I only did front-end template language coding, HTML and CSS. I never touched a database. And deployment back then was like taking a file, your FTP, And then you put the file on the server. It was basically taking things live process. I've never worked in a front-end framework. I've never touched a database. I've never ran a SQL query. People think like, oh, it's because Teresa's been an engineer. I can't explain to you how much of these engineering skills I did not have. That's why I was asking. So how much of the skills that you already had? How many of the things are familiar? because my kind of engineering background is slightly more engineering background maybe because I've really studied it was still really front-end heavy what we did back then PHP was around so that's what we use a lot and we developed we did a bit of compiler building and stuff like that so I looked into some back-end stuff basically but that was ages ago and so many things have changed ever since sometimes it helps when AI does stuff these days that I am like hmm maybe I should ask if Claude has done this particular step or if he's so it helps me with debugging sometimes but it's not that it's a transferable skill here's what i will share i i'm not going to undervalue what i got from my stanford computer science classes because here's what i did learn even in the very beginner classes they instill from day one this concept of elegant code it's all about how do you take a hard problem decompose it create code that's maintainable that's reusable and so like i do think i have engineering skills that i learned from my undergraduate degree like and i learned this from multiple perspectives so i didn't just get that from my computer science classes i also had to take philosophy classes where i learned how to deconstruct an argument in a very logical way right and so i'm not poo-pooing my education i actually think the reason why i can learn these skills is because of that foundation But I just want to be really clear. I didn't work. I had a lot of gaps. And the reason why I was able to do what I've been doing is because Claude can fill those gaps. Yeah, because you never stop learning. I think this is what both of us do. We never stop learning. Through our entire career, we always, whenever we feel slightly comfortable, we go look for our next project or our next, yeah. thing that we want to learn or get better at. And we're quite good in learning new things and picking up new skills. And I think that is what is a super helpful skill right now. So whoever's listening, practice to get better at something. I think this is what helps you to learn something quicker and not being kind of thrown off by something coming up like AI. There's a couple things. I think that it's a learning by doing, not learning by reading. I read a lot. Don't get me wrong, I read a lot. But what I love about LLMs now is you can just be like, I want to do this thing I have no idea how, set me on the right path. And then you get started and then you tackle another thing you don't know how to do and you can just do that. And so I want to tie this back to our last episode. We talked about the myth of product builders and not everybody has to be a product builder. I'm not sharing my experience because I think everybody needs to become an AI engineer. I actually wanted to share this because I've been on this path where after I broke my ankle, I decided I was just going to do whatever was of most interest to me. And I've been pulling this thread. And it turns out the more I pull the thread, the more of an AI engineer I become. And that's a little bit surprising to me. Like I love product. I love design. I loved being a coach. I loved teaching. And I'm just finding myself wanting to be an AI engineer all day, every day. And it was literally just finding, like not being afraid of what I don't know and just following the thing that like is fun for me. Yeah, where passion led you. Yeah. And so it's not, the takeaway isn't, oh, everybody should be an AI engineer. I think the takeaway is, One of the really powerful things of this new technology is it helps us learn things we don't know how to do. It helps us create things we don't know how to create. It helps us build skills in areas we wish we had always had. I always had engineering insecurities. To work as a front-end engineer and know nothing about databases, every job I had, I thought I was going to be discovered as a phony. I always had these limiting beliefs about engineering, and it turns out I don't anymore. Like I know anything that I don't know how to do, Claude will teach me how to do. And Claude is infinitely patient and I have no problem shamelessly asking a million questions. And like I said, for TeresaBot, I put the whole conversation on the web. So if you want to see like the things I had to ask about, like I know what embeddings are, but I don't know how an embeddings database works. it's fine to not know those things and you can still push forward. So I think like whatever it is you're excited about, it doesn't have to be AI engineering. I think the takeaway is you can just keep pulling on this thread. And for me, that's just super fun. And you might be surprised by where you land. I love that message. Thank you so much, Teresa. Amazing. Thanks for sharing your learnings. Thanks, Petra.

What happens when a product leader accidentally becomes an AI engineer? In this episode, Teresa Torres shares how she went from occasional tinkerer to spending 60% of her time doing real engineering work — building AI-powered tools for continuous discovery, forming a licensing partnership with Vistaly, and quietly constructing "Teresa Bot," an AI discovery coach trained on everything she's ever written.

Teresa and Petra dig into what AI engineering actually looks like in practice: context engineering, prompt writing, RAG, observability, evals, and why Teresa thinks product managers who want to do great discovery might secretly be data scientists at heart. They also bust the myth that you need a strong engineering background to build in this space — and make the case that the most important skill right now isn't coding. It's being willing to learn.

Topics Covered
✅ Teresa's accidental path into AI engineering
✅ Her partnership with Vistaly
✅ Building "Teresa Bot"
✅ How she learned — and how you can too
✅ Discovery skills transfer directly to AI engineering
✅ On her engineering background (and why it's not what you think)
✅ The real takeaway

Key Quotes
"I know anything that I don't know how to do, Claude will teach me how to do. And Claude is infinitely patient." — Teresa Torres
"I don't want to build all that stuff. I don't really want to be a software company. I'm almost set up like an AI researcher." — Teresa Torres
"The moment I learned more about data science, all of my discovery work became so different." — Petra Wille

Resources & Links:
Follow Teresa Torres: https://ProductTalk.org 
Follow Petra Wille: https://Petra-Wille.com

Mentioned in this episode: 
Behind the Scenes: Building the Product Talk Interview Coach blog by Teresa: https://www.producttalk.org/customer-interview-coach/
Claude Code: https://www.claude.com/product/claude-code
Vistaly, an opportunity solution tree software partnering with Teresa: https://vistaly.com/
Opportunity Solution Trees: Visualize Your Discovery to Stay Aligned and Drive Outcomes by Teresa: https://www.producttalk.org/opportunity-solution-trees/
Product Talk Academy by Teresa: https://learn.producttalk.org/
Just Now Possible Podcast by Teresa, which is a podcast on teams building AI products: https://www.producttalk.org/just-now-possible/
Vibe Coding Best Practices: Avoid the Doom Loop with Planning and Code Reviews blog by Teresa which includes the full Claude conversation used to build Teresa Bot: https://www.producttalk.org/vibe-coding-best-practices/
AI Evals for Engineers and PMs course by Hamel Husain and Shreya Shankar (get 35% off through Teresa’s link) on Maven: https://maven.com/parlance-labs/evals?promoCode=torres-35
CDH Membership (includes Slack community access with Teresa and other continuous discovery practitioners): https://www.producttalk.org/product-catalog-cdh-membership/
Previous All Things Product episode: Product Builder Myth: 

Have thoughts on this episode? Leave a comment below.
#AllThingsProduct