How to Actually Build AI Agent Loops?

Hey everyone, this is Sean. So today, we're going to go through probably the biggest buzzwords in AI agent system recently. Agent Harness, Loop Engineering, LNOPS, which stands for Large Language Models Operations, EVAL, which stands for Evaluation System for AI agents. And these things become popular or become viral on the internet, not because they're just some really complicated concepts. Instead, they're actually very simple. And I believe that simple building blocks will actually help us build the biggest architecture in the world that will function like an intelligent system. Let's walk through this step by step, and it does not matter if you're technical or not. We're gonna go through this, and we'll make sure that you're equipped with the right knowledge for prompting your way through building such a system in the future. Let's jump in and get started. For those of you who have watched my previous video on AI agent memories, you're probably already familiar with this chart. This is an AI agent run, which means that it takes an input from A user prompt, for example, you're asking ChatGPT or DeepSeek a question and say, hey, when was Sam Altman fired from OpenAI? And then it's going to go through an entire run, but the end goal is that you want to get a response. This is actually ephemeral, which means that there's no memory in this at all. We're sending that question, when was Sam Altman fired? And any chat history that's currently in the chat, for example, maybe we had some conversations before that, which, for example, could be, you should talk to me like Elon Musk grilling on Sam Altman because they don't like each other. And then these things will be fed into the... thing called a working memory or a context RAM in this video we probably won't dive too much in depth into the memory system because there's a previous video talking about it already but I'll just quickly go through it and then we'll introduce the concept of what a harness means when you have this kind of short-term working memory there will be an LLM or a large language model which performs as a question and answer agent and at the end you're gonna get a reply but the problem with a simple agent run was simply just the question current chat history and system prompt is that The memory is very short term, but when you run an AI agent system, sometimes we need extra memories. For example, how should the agent respond to the person? A procedural memory is exactly that. It basically tells the agent how to act and what are some of the instructions for the skill. We might also want the agent to know some durable facts about this context. For example, I might want to compare my own early stage startup journey with Sam Altman's early startup journey. We need this agent to have a memory of who I am, which in this context would be a durable facts. or a semantic memory who Sean is what did he build in the past these kind of things became a fact that you want your agent to know but they're not publicly available if you're not famous because the AI model won't be trained on such information yet but if you're famous already you can skip this they already know who you are and another thing we need is called episodic memory and they include things like the past events or past chat history that does not exist in this current conversation for example i might suddenly be wondering when was the last time i was preparing for a job application and can we retrieve that information and match you know if we can get a job in chat gpt so these things will be retrieved from this thing called an episodic memory which is basically a time series of the previous conversations or previous triggers that happen if you have a more complex system So for those of you who have watched my previous memory agent system design, you might be wondering, Sean, why are you repeating all of these things? And that is because if you think about the entire thing that we just covered in the past few minutes, we're really stating the one fact that a large language model can't do these things by itself. It's like a really powerful brain that knows everything about humanity, everything about science, anything that happened in human or biology history. But... it does not know you with you or the software who's running this ai agent system a large language model has no clue with how you want it to perform this is why the concept called harness becomes really important in this what harness means literally is that it's a set of harness tools that you use to control a horse when you're doing a horse riding imagine this large language model is a horse right this horse is very powerful they can run around but if you don't have a good set of tools to ride this horse you could just get heard you might go anywhere you might go somewhere random if you're in a war you don't want that to happen and that's why we're doing all of these to make sure we have good control over this large language model and make sure we're utilizing it at its maximum potential that's why in addition to just the question or use a prompt and getting the reply and we're feeding them all in as a working memory which can be enhanced by these three memories we just talked about and in order for these three memories to actually work there's a bit more details and they're all included in harness remember harness means we're building this agent framework to control this large language model so that it works the way we want for those of you who study statistics or machine learning you would understand that a large language model is actually predicting the probability of the next word that it should spit out when everything comes with probability there's randomness in it but when we solve problems we sometimes don't want too much randomness so that's why we need to have a good control this technology now let's continue to finish this harness there are lots of tools in the market that's already quite useful for example you could try tools like LangGraph, Lanchain or Pydantic and there are many others in this video we won't dive too much in depth into that and we're gonna finish building up this harness before we move on to the next topic so again for this agent to work properly we need this memory system to work but this memory system needs an update system because memory doesn't just exist or pop up from nowhere you need to constantly update it that's why we need a database to restore all these memories so that when the agent is running and this agent run it knows where to retrieve these memories so whenever you see an icon like this this is a database okay procedural memory is basically remember it's about instructions right it's about how the agent should be acting it's like with a harness on the horse you want the horse to ride faster or slower normally these are just files or text and that's why you probably heard of this buzzword called skills skill is basically a piece of text in a markdown file that you feed into ai agent like clock code but if you want to harness the system well just having files and text is not enough and they're stored in databases say like aws superbase google cloud you know azure all these kind of places or you can set up your own server at home if you want but that's just too expensive you don't want to do that and in order for this harness to work properly you also need to figure out how to store the memories okay so for example the episodic memory is the time series of the events that happen or the previous chat history again the way we store it is actually very simple you just track every single thing that happened and then it's going to become like a very long list of things that happen in history with time stamps durable fact is a different story you can either input it yourself or you want the system to sort of automatically evolve over time and the way for it evolve is that you want to consolidate some of the conversations into these semantic memory if i'm running a d2c brand e-commerce company perhaps my customers have talked to my customer service agents for a million times about how do I get reimbursed if this product does not work. You want to consolidate these conversations and distill them as a fact into the semantic memory. And that's why here we have a little gate here. If your brand has a million people purchasing products from it, let's say your Alibaba or Amazon, it just doesn't make sense. And it's very expensive. So from a harness perspective, you want to be smart about this and you want the system to be automatic. And in a simple way, it's probably like maybe consolidate these time ordered events after every say 2000 conversations because have a million customers and they can feed these things into a summarizer agent which is another large language model harness you can define the system prompt in this one you can probably feed it with some memories too you can configure different models maybe it could be a cheaper model because you're feeding too much text into it so the context window is too big and probably these are very expensive so you can use cheaper open source models if you want to having such a mechanism allows you to consistently update this memory system the data should be coming from the previous large language model replies again let's review how this harness works a user sent a prompt where in one agent runtime with the current chat history and how the agent should be performing the system prompt we're preparing a working memory for this AI agent to be able to answer a question and after every single time it answered a question it will send these messages to this database and then this database is basically feeding back to this working memory every single time when a question is checking a relevant context and at the same time because this database is too big sometimes you want to consolidate them into some summarized information or distilled facts so that they're stored properly in a semantic memory so that the retrieval of such memories is just faster i know we talked about retrieval a lot and that's just another buzzword called rag which is retrieval augmented generations i also have a few videos explaining what rags are feel free to watch them there's a little bit of difference between how i retrieve from semantic memory and episodic memory for semantic memory it's just rags because these are just facts and text or files, right? But then for episodic memory, remember this is a time series. Let's say we're still in this e-commerce store, right? The user question could be like, what were the previous 10 conversations that we had with this specific customer from the US? And then you might just need a SQL query to query something that's pretty recent from this episodic memory. But if your question is like, what were my previous 20 conversations that have customer complaints on the quality of the products and our agent did not successfully resolve? With such a question, you not only need a SQL query, which is just capturing the data events in a data table, you also want to do some semantic search. And that's why here RAG is important, because it's checking for relevant information for you. You don't want the entire 2,000 messages. You want that 20 messages out of these 2,000 that are exactly relevant to what you want. And because these complaints are in text, we need to do some retrieval augmented generation to match the semantic meanings. between text and the user prompt so that you're fetching the right context for the working memory. By now, this is probably a fast walkthrough of the memory system again, but we're just thinking about it from a Harness perspective in this video. And remember for Harness, we're training this horse of LLM to run autonomously without having too much randomness, okay? There's another piece of it that's quite important, which is the agent might not only just read the memory, it might also do some tasks or call some tools when an agent calls tools it might not necessarily be just one time call it could be multiple times of calls for instance let's say this ai agent has a bunch of authentic tools such as help me schedule a meeting help me read or write my custom relationship data from the crm system or help me fetch the payment information say from stripe or alipay and here's something we should be careful about if we give this horse or this llm technology full power it could just continuously do this forever right or it might not even know what's the right time to stop or what is the right tool cause it should make when is the end point to decide okay this response is good enough let's move on to reply that's why we have this mechanism called end loop guardrails yes now we're talking about loop engineering one of the biggest buzzwords in the recent few months a loop is part of harness why because a loop is also helping us to control technology to make sure it runs the way we want it to run an example that could be helpful for you is that let's say the custom prompt is help me find out what customers are complaining about our products what are some of the follow-ups we could do in order to win them back and if they're asking for reimbursement have we done the reimbursement or not if not can we do that this is probably a series of questions but sometimes we just dump all of these things into AI agent Okay. And after you have this prompt, this LLM agent needs to decide, okay, what are some of the tools that could be helpful for me to finish this task? Loop here is basically an architectural thinking of when is good enough so that we stop and give the user or the business owner a reply. Okay. So what might happen here is that the LLM agent is doing a bunch of tool calls. It's doing some thinking and saying, let me read. from our customer relationship management tool, like Salesforce, HubSpot, or Automanus, and then it's gonna find out, okay, there were 30 customer complaints in the past two months. 12 of them have got reimbursement, and the other eight have not got reimbursement. So after the first initial fetch, it's probably responding to the ALI agent, right? And the agent will be probably thinking, okay, the task or the ask is that, can we follow up with some of those who did not get the reimbursement right so and then it probably like just make another tool calls and be like hey let's schedule a meeting with those customers who did not get a reimbursement which are the eight of them if we go a little bit more advanced we even just use the reimbursement trigger on stripe or Alipay to refund the customer can you see that this is a loop until we finish the task but of course this is like a case-by-case situation it really depends on what your task is how you build the system so there's no one solution fits all Here, I'm just explaining what a loop is. The very, very essential part of this loop is that it needs to know when it should stop. That's why we need this end loop guardrails. The guardrails could just simply be the task is done, and perhaps when the agent was doing the planning, it should confirm with the user what is a good ending point. It might clarify with you, is this what you want, reimbursing the other eight people, or should I just tell you who they are, and then you will follow up later. These are two different decisions you can make. And after you make, you're basically telling the agent loop that there is an ending scenario. Another good example I saw today is that you know when you're doing coding and ClockCode can just always pop up some windows and ask you for permissions right so a good way to use a loop engineering here is that you can set up a loop or set up a hook in ClockCode and telling it that you should always send me a notification on my laptop if you are pending on some permissions from me otherwise if I'm watching YouTube and then when I come back 30 minutes later realized that clock code is stuck in that one permission like 25 minutes ago that would be a waste of my time okay so you can set up a loop like this to make sure that there's a way to send you notification pop-ups so that you know the loop has ended or you needs your input again are you guys still with me good so by far we have covered AI agent run with a memory system with a loop engineering around the large language model agent which has a trigger to end the loop so that it sends a reply to the user And basically this whole thing is an AI agent harness system. What's next is one of the other biggest buzzwords that Y Combinator always mentions, which is eval or LLM ops. Let's jump into it. But firstly, I want you to understand why do we need LLM ops here? Let's still look at the left-hand side with this harness system. The biggest problem here is that we don't know how well it's performing. And that's why we need a feedback loop to help us understand, is this agent actually performing properly? right for my business or for my use case and can i continuously get feedback on how do we fix it and actually fix it ourselves okay and when we say fix it a simple way to understand it is that can we have a better system prompt can we have a better large language model configurations is there something we should change for how we retrieve the ai agent memories these are kind of things that we can continue to iterate but in order to iterate to make sure this system runs properly we need a way to evaluate it diagnose problems solve the problems until it's a healthy and well-performing system and that is called large language model operations system LLM ops so again in order to understand this properly we need to come back to what an agent run is so an agent run you can simply understand it as a user question is sent to a large language model and they get a reply That is one agent run. But in this agent run, the agent tool calling could happen multiple times. That does not matter. We're just talking about from a user input to a response from agent perspective. That's one agent run. And then we're going to introduce this system called a tracing system. So every agent run... we should trace like a tree of events that happened and there are lots of tools that can help you with that it could be landfuse could be landsmith etc etc a tree of events could be like what did the person actually ask what retrievals did the model actually retrieve how many times did the large language model actually call the tools and how was the tool usage how was the response time right how long did it take for this entire system to run for checking latencies and how many tokens have we used when we do these tool calls agent run you know doing this retrieval augmented generation these kind of things so trace is helping us to track events basically and that's the first step this is the step first to collect data and these data will be used for the following two purposes was it a good system run and was it healthy which corresponds to evaluation system we can probably use large language model as a judge here to give us a score on how well it performed for example if the task had something to do with scheduled meetings, did the meeting actually trigger? How long was the response for an agent to reply to a question? Was it 20 seconds or was it two milliseconds? And also things like how many tokens have we used? These two are basically in the same system. You can write it as deterministic code, you can use an AI agent to do it, but this is like part of the procedure, which is helping us to understand, was this a healthy system and was it a good system? And after that, we're gonna diagnose, okay, where and why something was broken. for example the meeting scheduling event was never triggered why was that right we won't understand why was that and we could probably feed that into a coding agent in Claude to sort of deep dive into it or you know if the latency is 20 seconds instead of two milliseconds something's wrong maybe one of the tool call is taking too much time maybe the working memory is too large so that the response time for a large language model to a memory retrieval is just taking too much time maybe not every single question Requires a retrieval from all these gigantic memory system. Maybe you're just asking a simple question be like when was my birthday? When was open AI started and these kind of information probably don't need to do a ton of retrieval the model itself already knows so you basically want this system to provide a dashboard for you to understand the metrics and Then with these metrics you can diagnose what is going on and then we're gonna have a little gate here which is if the evaluation system passed well you can define the rules we can either ship some very simple fix have a new version of the prompt or update the model configuration you know some tool changes or the parameters for retrievals the llm ops will feed the improved system prompt and the configuration of the model back to this agent run system when then one llm ops loop is finished if let's say something is deeply broken right We cannot just simply ship the latest version of the prompt. Then we should go fix the bug, rerun the agent run, resend the question, and then retrace the events, and then redo this evaluation system in this LLM ops architecture. So now let's zoom out and look at this chart one more time. We covered what an AI agent run is, we covered how it would retrieve information from memories, and we understood how an LLM agent would ask questions. would call tools to help it finish the task in a loop and it knows when to stop the loop so that we can get the reply and this whole thing is a set of harness tools that we're controlling this horse this technology to run in the right direction okay to do the right task and at the same time we have like a health checking system or evaluation system to understand how every single run is being traced is being observed and how do we diagnose the problems and fix some problems and ship the latest updates the prompt about the model configuration about all these parameters or knobs that needs to be updated so that this system will be an autonomous system that will just self evolve and grow over time I really hope this was helpful let me know what you think and you have any questions you can always reach out to me

If you are still typing prompts one by one, you are acting as the execution layer for AI. In this video, we breakdown the exact "Loop Engineering" framework used by top engineers to build autonomous, self-correcting agent loops that handle bulk tasks, verify their own code, and fix their own errors while you sleep.

We move past complex multi-agent hype and dive straight into the 5-Beat Framework you can use to build reliable agentic systems today.

Here’s exactly how he does it:

→ Give it memory
→ Harness the loop
→ Trace what happened
→ Judge the result
→ Fix what’s broken
→ Ship


🕒 Video Timestamps
0:00 - The Death of Prompt Engineering
1:42 - What is Loop Engineering? (Boris Cherny's Framework)
3:15 - The 5 Beats of an Agent Loop
5:40 - Coding Step 1: Defining the Measurable Goal
7:22 - Coding Step 2: Setting up Environmental State & Memory
9:45 - Building the Self-Check Execution Loop
12:10 - Preventing Token Runaway & Infinite Retries
14:30 - Run it Live: AI working while you sleep

When NOT to use a Loop: One-off tasks don't need this overhead. Loops earn their setup cost on high-volume, structural, or repetitive workflows.

If you are ready to transition from a prompt engineer to a loop engineer, make sure to like, subscribe, and drop a comment with the first loop you're building!

#AIAgents #LoopEngineering #ClaudeCode #LangGraph #Anthropic #Python #SoftwareEngineering #AIAutomation