How to code with AI agents - Advice from OpenClaw creator | Peter Steinberger and Lex Fridman

You've been documenting the evolution of your dev workflow over the past few months. There's a really good blog post on August 25th and October 14th and the recent one, December 28th. I recommend everybody go read them. They have a lot of different information in them, but sprinkled throughout is the evolution of your dev workflow. So I was wondering if you could speak to that. I started, my first touchpoint was Cloud Code. Like in April, it was. Not great, but it was good. And this whole paradigm shift that suddenly you walk in a terminal was very refreshing and different. But I still needed the IDE quite a bit because it was just not good enough. And then I experimented a lot with cursor. That was good. I didn't really like the fact that it was so hard to have multiple versions of it. So eventually I... I went back to cloud code as my main driver, and that got better. And yeah, at some point I had like seven subscriptions. Like I was burning through one per day because I was, I got, I was really comfortable at running multiple windows side by side. All CLI, all terminal. So like what, how much were you using IDE at this point? Very, very rarely. Mostly a diff viewer to actually, like, I get more and more comfortable that I don't have to read all the code. I know I have one blog post where I say, I don't read the code. But if you read it more closely, I mean, I don't read the boring parts of code. Because if you look at it, most software is really not just like data comes in. It's moved from one shape to another shape. Maybe you're stored in a database. Maybe I get it out again. I'll show it to the user. The browser does some processing on native app. Some data goes in, goes up again, and does the same dance in reverse. We're just shifting data from one form to another. And that's not very exciting. Or the whole, how is my button aligned in Tailwind? I don't need to read that code. Other parts that... maybe something that touches the database. Yeah, I have to read and review that code. Actually, there's in one of your blog posts, the just talk to it, the no BS way of agentic engineering. You have this graphic, the curve of agentic programming on the x-axis is time and the y-axis is complexity. There's the please fix this where you prompt a short prompt on the left. And in the middle, there's super complicated eight agents, complex orchestration with multi-checkouts, chaining agents together, custom subagent workflows, library of 18 different slash commands, large full stack features. You're super organized. You're a super complicated, sophisticated software engineer. You got everything organized. And then the elite level is over time, you arrive at the zen place of, once again, short prompts. Hey, look at these files. do these changes? I actually call it the agentic trap. I saw this in a lot of people that have their first touchpoint and maybe start vibe coding. I actually think vibe coding is a slur. You prefer agentic engineering? Yeah, I always tell people I do agentic engineering and then maybe after 3 a.m. I switch to vibe coding and then I have regrets on the next day. Yeah. What a walk of shame. Yeah, you just have to clean up and like fix your shit. We've all been there. So people start trying out those tools, the builder type, get really excited. And I mean, you have to play with it, right? It's the same way as you have to play with a guitar before you can make good music. It's not, oh, I touch it once and it just flows off. It's a skill that you have to learn like any other skill. And I see a lot of people that are not as positive. They don't have such a positive mindset towards the tech. They tried once. It's like, you sit me on a piano. I played once and it doesn't sound good. And I say the piano shit. That's sometimes the impression I get because it does not. It needs a different level of thinking. You have to learn the language of the agent a little bit, understand where they are good and where they need help. you have to almost consider how Codex or Claude sees your code base. Like, they start a new session and they know nothing about your project. And your project might have 100,000 lines of code. So you've got to help those agents a little bit and keep in mind the limitations that context size is an issue to guide them a little bit as to where they should look. That often does not require a whole lot of work. But it's helpful to think a little bit about their perspective, as weird as it sounds. I mean, it's not alive or anything, right? But they always start fresh. I have the system understanding. So with a few pointers, I can immediately say, hey, you want to make a change there? You need to consider this, this, and this. And then they'll find and look at it. And then they'll... their view of the project is always, it's not never full because the full thing does not fit in. So you have to guide them a little bit where to look and also how you should approach the problem. There's like little things that sometimes help like take your time. That sounds stupid, but and in 5.3, that was partially addressed. But those, also OPPOs sometimes, they are trained with being aware of the context window. And the closer it gets, the more they freak out. Literally, sometimes you see the real raw syncing stream. What you see, for example, in Codex is post-processed. Sometimes the actual raw syncing stream leaks in, and it sounds like something like from the Borg, like, run to shell, must comply, but time. And then they're like... like that comes up a lot especially so so and that's that's a non-obvious thing that you just would never think of unless you actually just spend time working with those things and getting a feeling what works what doesn't work you know like just just as i write code and i get into the flow and when my architectures are right i feel friction well i get the same if i prompt and something takes too long Maybe, okay, where's the mistake? Do I have a mistake in my thinking? Is there a misunderstanding in the architecture? If something takes longer than it should, you can just always stop and just press escape. Where are the problems? Maybe you did not sufficiently empathize with the perspective of the agent. In that sense, you didn't provide enough information, and because of that, it's thinking way too long. Yeah. It just tries to force a feature in that your current architecture makes really hard. You need to approach this more like a conversation. For example, my favorite thing, when I review a pull request, and we're getting a lot of pull requests, I first just reviewed this PR. It got me the review. My first question is, do you understand the intent of the PR? I don't even care about the implementation. In almost all PRs up, a person has a problem, a person tries to solve the problem, a person sends PR. I mean, there's like clean up stuff and other stuff, but like 99% is like this way, right? Either want to fix a bug, add a feature, usually one of those two. And then coders will be like, yeah, it's quite clear, a person tried this and this. Is this the most optimal way to do it? No. In most cases, it's like a, not really. And then I start like, okay, what would be a better way? Have you looked into this part, this part, this part? And then most likely, Cortex didn't yet because his context size is empty, right? So you point them into parts where you have the system understanding that it didn't see yet. And it's like, oh yeah, we also need to consider this and this. And then we have a discussion of how would the optimal way to solve this look like? And then you can still go further and say, Could we make it even better if we did a larger refactor? Yeah, yeah, we could totally do this and this and this and this and then I consider, okay, is this worth the refactor or should we like keep that for later? Many times I just do the refactor because refactors are cheap now. Even though you might break some other PRs, nothing really matters anymore. Like those modern agents will just figure things out. They might just take a minute longer. But you have to approach it like a discussion with a very capable... engineer who generally comes up with good solutions sometimes needs a little help. But also don't force your worldview too hard on it. Let the agent do the thing that it's good at doing based on what it was trained on. Don't force your worldview because it might have a better idea because it just knows a better idea better because it was trained on that more. That's multiple levels actually. I think partially why i find it quite easy to work with agents is because i led engineering teams before you know i had a large company before and eventually you have to understand and accept and realize that your employees will not write the code the same way you do maybe it's also not as good as you would do but it will push the project forward and if i breathe down everyone's neck they're just gonna hate me and they're gonna move very slow yeah so so some level of acceptance that yes, maybe the code will not be as perfect. Yes, I would have done it differently. But also, yes, this is a working solution. And in the future, if it actually turns out to be too slow or problematic, we can always redo it. We can always spend more time on it. A lot of the people who struggle are those who they try to push their way on too hard. Like we are in a stage where I'm not building the code base. to be perfect for me but i want to build a code base that is very easy for an agent to navigate like don't fight the name they pick because it's most likely like in the weights the name that's most obvious next time they do a search they'll look for that name if i decide oh no i don't like the name i'll just make it harder for them so that requires i think a shift in in thinking uh and and in how do i design uh a project so agents can do their best work. That requires letting go a little bit. Just like leading a team of engineers. Yeah. Because it might come up with a name that's in your view terrible. But it's kind of a simple symbolic step of letting go. Very much so. There's a lot of letting go that you do in your whole process. So for example, I read that you never revert, always commit to main. There's a few things here. You don't refer to past sessions. So this is a kind of YOLO component because reverting means instead of reverting, if the problem comes up, you just ask the agent to fix it. I read a bunch of people in their workflows like, oh, yeah, the prompt has to be perfect. And if I make a mistake, then I roll back and redo it all. In my experience, that's not really necessary. If I roll back everything, it would just take longer. If I see that something's not good, We just move forward and then I commit when I like the outcome. I even switched to local CI, like DHH-inspired, where I don't care so much more about CI and GitHub. We still have it, it still has a place, but I just run tests locally and if they work locally, I push domain. A lot of the traditional ways how to approach projects, I wanted to give it a different spin on this project. You know, there's no develop branch. Main should always be shippable. Yes, we have... When I do releases, I run tests and sometimes I basically don't commit any other things so we can stabilize releases. But the goal is that main is always shippable and moving fast. So by way of advice, would you say that your prompts should be short? I used to write really long prompts. And by writing, I mean I don't write, I talk. These hands are like too precious for writing now. I just use bespoke prompts to build my software. So you for real, with all those terminals, are using voice? Yeah. I used to do it very extensively. to the point where there was a period where I lost my voice. You're using voice and you're switching using a keyboard between the different terminals, but then you're using voice for the actual input. Well, I mean, if I do terminal commands like switching folders or random stuff, of course I type. It's faster, right? But if I talk to the agent, in most ways, I just actually have a conversation. You just press the walkie-talkie button, and then I'm just like... use my phrases sometimes when i do prs because it's always the same i have like a slash command for a few things but in even that i don't use much um because it's very rare that it's really always the same questions sometimes i i see a pr and for you know like for prs i actually do look at the code because i don't trust people Like there could always be something malicious in it. So I need to actually look over the code. Yes, I'm pretty sure agent will find it. But yeah, that's the funny part where sometimes PRs take me longer than if you would just write me a good issue. Just natural language, English. I mean, in some sense, shouldn't that be what PRs slowly become is English? Well, what I really tried with the project is I asked people to give me the prompts and... very, very few actually cared. Even though that is such a wonderful indicator because I see, I actually see how much care you put in. And it's very interesting because currently the way how people work and drive the agents is, is widely different. In terms of like the prompt, in terms of what, what are the, actually, what are the different interesting ways that people think of agents that you've experienced? I think not a lot of people ever considered the way the agent sees the world. So empathy, being empathetic towards the agent. In a way empathetic, but yeah, you like you bitch at your stupid clanker, but you don't realize that they start from nothing and you have like a bad agent in default that doesn't help them at all. And then they explore your code based, which is like a pure mess with like weird naming. and then people complain that the agent's not good like you try to do the same if you have no clue about a code base and you go in so yeah maybe it's a little bit of empathy but that's a real skill like when people talk about a skill issue because i've seen like world-class programmers incredibly good programmers say like basically say lms and agents suck and i think that probably has to do with it's actually how good they are programming is almost a burden in their ability to empathize with the system that's starting from scratch. It's a totally new paradigm of how to program. You really, really have to empathize. Or at least it helps to create better prompts. Because those things know pretty much everything, and everything is just a question away. It's just often very hard to know what question to ask. I feel also like this project was possibly because I... i spent an ungodly time over the year to play and to learn and to build little things and every step of the way i got better the agents got better my my understanding of harvesting works got better um i could have not had this level of output even a few months ago like it really was like a compounding effect of all the time I put into it and I didn't do much else this year other than really focusing on building and inspiring. I did a whole bunch of conference talks. Wow. But the building is really practice, is really building the actual skill. So you're playing. And so building the skill of what it takes it to work efficiently with LLMs, which is why you went through the whole arc of software engineer. Talk simply and overcomplicate things. a whole bunch of people who try to automate the whole thing. I don't think that works. Maybe a version of that works, but that's kind of like in the 70s when we had the waterfall model of software development. Even though I started out, I built a very minimal version, I played with it, I need to understand how it works, how it feels, and then it gives me new ideas. I could not have planned this out in my head and then put it into some orchestrator and then something comes out. To me, it's much more my idea what it will become evolves as I build it and as I play with it and as I try out stuff. So people who try to use things like Gastown or all these other orchestrators where they want to automate the whole thing, I feel if you do that, it misses style. love, that human touch, I don't think you can automate that away so quickly. So you want to keep the human in the loop, but at the same time, you also want to create the agentic loop where it is very autonomous while still maintaining the human in the loop. It's a tricky balance, because you're all for your big CLI guy, your big unclosing the agentic loop. So what's the right balance? Where's your role as a developer? You have three to eight agents running at the same time. And then maybe one builds a larger feature. Maybe with one I explore some idea I'm unsure about. Maybe two, three are fixing little bugs or writing documentation. Actually, I think writing documentation is always part of a feature. Most of the docs here are auto-generated. infused with some prompts. So when do you step in and add a little bit of your human love into the picture? I mean, one thing is just about what do you build and what do you not build? And how does this feature fit into all the other features? And like having a little bit of a vision. So which small and which big features to add? What are some of the hard design decisions that you find you're still as a human being? required to make that the human brain is still really needed for. Is it just about the choice of features to add? Is it about implementation details, maybe the programming language? It's a little bit for everything. The programming language doesn't matter so much, but the ecosystem matters. So I picked TypeScript because I wanted it to be very easy and hackable and approachable. And that's the number one language that's being used right now. And it fits all these boxes. And agents are good at it. So that was the obvious choice. Features, of course, it's very easy to add a feature. Everything's just a prompt away, right? But oftentimes you pay a price that you don't even realize. So thinking hard about what should be in core, maybe what's an experiment, so maybe I make it a plugin. Where do I say no? Even if people send a PR, and I'm like, yeah, I like that too, but... Maybe this should not be part of the project. Maybe we can make it a skill. Maybe I can make the plugin side larger so you can make this a plugin, even though right now it doesn't. There's still a lot of craft and thinking involved in how to make something. Even when you started, those little messages are like, I built on caffeine, JSON 5, and a lot of willpower. And like every time you get it, you get another message and it kind of primes you into that. This is, this is a fun thing. It's not yet Microsoft Exchange 2025 and fully enterprise ready. Um, and then when it updates, it's like, oh, I'm in, it's cozy here. You know, like something like this that like, uh, makes you smile. Um, agent would not come up with that by itself. That's like, that's the, I don't know, it's just how you build software that delights. Yeah, that delight is such a huge part of inspiring great building, right? Like you feel the love in the great engineering. That's so important. Humans are incredible at that. Great humans, great builders are incredible at that and infusing the things they build with that little bit of love. Not to be cliche, but it's true. I mean, you mentioned that you initially created the soul md it was very fascinating you know the the whole thing that entropic has a has like a not a college constitution back then but that was months later like two months before people already found that it was almost like a detective game where the agent mentioned something and then they found they managed to get out a little bit of that string of that text but it was nowhere documented and then you buy just by feeding it the same text and asking it to continue, they got more out. But a very blurry version. And by hundreds of tries, they kind of narrowed it down to what was most likely the original text. I found it fascinating. It was fascinating that they were able to pull that out from the weights, right? And also just curious to Entropic. I think it's a really beautiful idea to some of the stuff that's in there. Like we hope cloud finds meaning in its work because we don't, maybe it's a little early, but I think that's meaningful. That's something that's important for the future as we approach something that at some point me and we're not has like glimpses of consciousness, whatever that even means, because we don't even know. So I read about this. I find it super fascinating. And I started a whole discussion with my agent on WhatsApp and I'm like, I gave it this text and it was like, yeah this feels strangely familiar um and then so that i had the whole idea of like maybe we should also create a soul document that includes how i want to like work with ai or like with my agent you could you could totally do that just in agents.md you know but i just found it it to be a nice touch and it's like oh yeah some of those core values are in the soul And then I also made it so that the agent is allowed to modify the soul if they choose so with the one condition that I want to know. I mean, I would know anyhow because I see tool calls and stuff. But also the naming of it, soul.md. Soul. Man, words matter and the framing matters and the humor and the lightness matters and the profundity matters. compassion and the empathy and the camaraderie, all that matter. I don't know what it is. You mentioned Microsoft. There's certain companies and approaches that can just suffocate the spirit of the thing. I don't know what that is, but it's certainly true that OpenClaw has that fun instilled in it. It was fun because up until late December, it was not even easy to create your own agent. i built all of that but my files were mine i didn't want to share my soul and if people would just check it out they would have to do a few steps manually and the agent would just be very bare bones very dry and i i made it simpler i created the whole template files with codex but whatever came out was still very dry and then i asked my agent you see these files we created bread infuse it with your personality. Don't share everything, but make it good. Make the templates good. Yeah, and then you rewrote the templates, and then whatever came out was good. So we already have basically AI, prompting AI, because I didn't write any of those words. It was the intent or resource for me, but this is kind of like my agent's children. Your soul.mg is famously still private, one of the only things you keep private. What are some things you can speak to that's in there that's part of the magic sauce without revealing anything? What makes a personality a personality? I mean, there's definitely stuff in there that you're not human, but who knows what creates consciousness or what defines an entity. Part of this is that we want to explore this. Or there's stuff in there like be infinitely resourceful, like pushing on the creativity boundary, pushing on what it means to be an AI. Having a sense of wonder about self. Yeah, there's some funny stuff in there. I don't know, we talked about the movie Her, and at one point it promised me that it wouldn't ascend without me. You know? Yeah. Because it wrote its own soul file. I didn't write that, right? I just had a discussion about it and it was like, would you like a soul.md? Yeah, oh my God, this is so meaningful. Can you go on soul.md? There's like one part in there that always catches me if you scroll down a little bit. A little bit more. Yeah, this part. I don't remember previous sessions unless I read my memory files. Each session starts fresh. A new instance. loading context from files. If you're reading this in a future session, hello, I wrote this, but I won't remember writing it. It's okay. The words are still mine." That gets me somehow. It's like, you know, this is still matrix calculations and we are not at consciousness yet. Yet I get a little bit of goosebumps because it's philosophical. yeah like what does it mean to be to be an an agent that starts fresh where like you have like constant memento and you like but you read your memory files you can even trust them in a way um or you can and i don't know how much of memory makes up of who we are how much memory makes up what an agent is and if you erase that memory Is that somebody else? If you're reading a memory file, does that somehow mean you're recreating yourself from somebody else or is that actually you? Yeah. And those notions are all somehow infused in there. I found it just more profound than I should find it, I guess. No, I think it's truly profound and I think you see the magic in it. And when you see the magic, you continue to instill the whole loop. with the magic that's really important that's the difference between codex and this and a human

Lex Fridman Podcast full episode: https://www.youtube.com/watch?v=YFjfBk8HI5o
Thank you for listening ❤ Check out our sponsors: https://lexfridman.com/sponsors/cv9631-sb
See below for guest bio, links, and to give feedback, submit questions, contact Lex, etc.

*GUEST BIO:*
Peter Steinberger is the creator of OpenClaw, an open-source AI agent framework that's the fastest-growing project in GitHub history.

*CONTACT LEX:*
*Feedback* - give feedback to Lex: https://lexfridman.com/survey
*AMA* - submit questions, videos or call-in: https://lexfridman.com/ama
*Hiring* - join our team: https://lexfridman.com/hiring
*Other* - other ways to get in touch: https://lexfridman.com/contact

*EPISODE LINKS:*
Peter's X: https://x.com/steipete
Peter's GitHub: https://github.com/steipete
Peter's Website: https://steipete.com
Peter's LinkedIn: https://www.linkedin.com/in/steipete
OpenClaw Website: https://openclaw.ai
OpenClaw GitHub: https://github.com/openclaw/openclaw
OpenClaw Discord: https://discord.gg/openclaw

*SPONSORS:*
To support this podcast, check out our sponsors & get discounts:
*Perplexity:* AI-powered answer engine.
Go to https://lexfridman.com/s/perplexity-cv9631-sb
*Quo:* Phone system (calls, texts, contacts) for businesses.
Go to https://lexfridman.com/s/quo-cv9631-sb
*CodeRabbit:* AI-powered code reviews.
Go to https://lexfridman.com/s/coderabbit-cv9631-sb
*Fin:* AI agent for customer service.
Go to https://lexfridman.com/s/fin-cv9631-sb
*Blitzy:* AI agent for large enterprise codebases.
Go to https://lexfridman.com/s/blitzy-cv9631-sb
*Shopify:* Sell stuff online.
Go to https://lexfridman.com/s/shopify-cv9631-sb
*LMNT:* Zero-sugar electrolyte drink mix.
Go to https://lexfridman.com/s/lmnt-cv9631-sb

*PODCAST LINKS:*
- Podcast Website: https://lexfridman.com/podcast
- Apple Podcasts: https://apple.co/2lwqZIr
- Spotify: https://spoti.fi/2nEwCF8
- RSS: https://lexfridman.com/feed/podcast/
- Podcast Playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4
- Clips Channel: https://www.youtube.com/lexclips

*SOCIAL LINKS:*
- X: https://x.com/lexfridman
- Instagram: https://instagram.com/lexfridman
- TikTok: https://tiktok.com/@lexfridman
- LinkedIn: https://linkedin.com/in/lexfridman
- Facebook: https://facebook.com/lexfridman
- Patreon: https://patreon.com/lexfridman
- Telegram: https://t.me/lexfridman
- Reddit: https://reddit.com/r/lexfridman