wtf is Loop Engineer & how to setup for real

Thanks HubSpot for sponsoring this video. It was about 1 a.m. yesterday and there were a whole bunch of PRs keep submitting to our code base. And that's not because we work extremely hard. It was all the different agent loops that is automobiles finding issues and picking up the work. And I even have this GoLoop has been running for past two days straight, where every day is outputting 20 to 40 extremely high quality pages that's driving traffic to my company without me looking at it. And this is what I want to talk about today, the loop engineer. It is the hardest thing everyone was talking about last week that you should no longer prompting the coding agent anymore. instead designing loops that atomic prompts agents and in this video i want to explain what loop engineer actually is what are core components and tips to making sure the loop actually works well as well as how my team has designed loop in a way that it actually compounds before we go deeper i want to quickly mention something useful especially if you haven't built any agents before because what i'm talking about today is the loop engineering harness shared memory can sounds a bit advanced but underneath all this the basic pattern is actually quite simple every agent still comes back to three things it brings the agent loop the memory layer and tool access and HubSpot has these free video courses that walk you through the foundation in very practical way it shows you how to build an agent from scratch using different type of tools including personal assistant agent that reads email and check your calendars and support style agent that connect to real business workflow and through those examples you get much better understanding about how the agent memory works how to effectively manage contacts So if you have never built an agent before, or you still trying to understand the difference between normal automation and actual AI agent, I think this great place to start because once you understand the basic pattern, what I'm talking about in this video becomes much easier to understand. I have put the link in the description below so you can check out for free. And thanks HubSpot for sponsoring this video. Now let's get back to what Loop Engineer actually is. So past one year, there are a lot of new terms that popping up. It might feel confusing, but in fact, each one of those terms. are a cluster of techniques that has been introduced for different level usage of large learning model. If we go back to 2023, when the GVT 3.504 API just show up, the majority of tasks we get a large learning model do is pretty simple. It's mostly the task completion. You give the API input and use large learning model to pretty much output text so it can be used for extracting structured data or writing blocks. But the nature of the model is undeterministic. And that's where the term prompt engineer show up. It was basically techniques about how to inject right context in the large learning model call. to steer the behavior like you can tell it always return tags in all caps then it write article in certain stuff but very quickly as we move into the mid 2024 that's where the model not only get much smarter but also way bigger contacts window back days the contact window is somewhere around 4 000. And when 128K tokens show up, it was mind-blowing advancement. Then Google just raised the bar to every model default have 1 million token context window. And the bigger context window here means the use case of how we use large-land model as it changed. We'll start building the system where the model is equipped with different tools like MCP. So it can decide what to do. And we include both the two call and two response as part of conversation. So the large-land model can continue this loop until it thinks the task is completed. And what this really changed is that the context window actually getting it up more and more as the model is capable to do more and more tasks and as we all know even though it has 1 million context window the effective window is somewhere between 128k to 200k whether you can feed the most relevant information within this context window directly impacts the agent's performance and this is where people start introduce a whole bunch of techniques around this like what to keep in the system problem that can trigger the prompt cache better and how to handle the long conversation regarding the compaction strategy. And also new concepts like SKU were introduced as a way to extend agents' capability without blow up the context window. But one thing really changed as we proceed to the end of 2025, which is we are getting model to do way longer and bigger tasks. From beginning of this year, it started becoming common that people are one-shot cloud code to finish 30 minutes, even two hours amount of work. And this is also a time where we start experimenting a lot about loops and workflows. In the beginning, people were trying rough loop, which is simple while loop, to run cloud infinitely with the same prompt. And later, cloud code introduced the concept of goal and also loops and recently workflow. And also things are pointing to one thing. We are getting models to do this kind of cross-session work. Meaning we no longer just getting one agent to finish the whole task. Instead, we can have multiple different agent sessions where each is handling one task and running in the loop until everything is finished. And this means we need a way to track the state or kind of file system across those different sessions so that each agent session can actually understand where things are at and continuously doing the work. And those are things actually happening outside agent runtime itself, but also including the environment where the agent is operating in. And this is where this agent harness concept was introduced. It was initially mentioned by V from LanChain. And the definition back then was pretty straightforward. Basically, harness means anything that is not model. And this is why the harness concept is so confusing because it just includes so many different things from the problem engineer and how you manage contacts as well as those orchestration logic and hooks. But one useful way always in my mind was there are two parts of optimization that can happen. One is agent loop itself, which can be cloud code or codex. If you're building your own PI agent, there's a whole bunch of optimization you can do. But all those techniques around how do you make sure when you give one task to the agent, it can finish the task pretty well. But on the other hand, there's a whole bunch of techniques that is not just about how to get agent complete a task well, but around how to get this whole agent existence, decide what should be worked on. And this outer part is actually what we currently talk about loop engineer. It's more like this kind of environment that you are setting up to both triggering the agent runtime, but also keep track of the state and logs so that it can continuously improve. And the reason this critical is that then it actually free you from prompting the agent itself. The agent can actually be a lot more autonomous and triggered by other different things. Like could be a cron job or it could be another agent. and even webhooks of like incidents that happen in your server. All those things can just trigger the agent getting to deliver meaningful piece of work without you involved. And this is a core mechanism for the loop engineer is that you will set up the right trigger so the agent can be wake up in violent time and scenario. And every time normally agent will do some sort of investigation and action which will produce a list of backlogs or ideas that the main agent can prioritize and assign tasks to others if needed. So next time you can review and learn. Let's take one example. Assume you are trying to create a loop for agent to handle the support. You can actually build a simple loop where every 30 minutes, the agent will be just wake up by the crown to review all support tickets, respond to ones that have been handled automatically, and also log the frictions and ideas. So you can pick that up as a product improvement later. And this loop itself is absolutely valuable. But what's even more powerful is what if it not only just logs those ideas and frictions. but actually trigger a coding agent to directly implement some of the ideas. They can actually monitor performance or even tell the customer those fix has been in place and monitor if any other people still experience those type of issues. And those two loops, they both provide huge value. The second one is a lot more powerful. And in my own experience, you can actually define multiple different loops that compound on each other if you can define a good logging system. So those are the loops that are actually happening in my own company. we have one support loop that every 30 minutes it will just trick the agent handling all support tickets and also log the frictions and ideas into one folder what we call signals signal is like a folder where it will capture either the product ideas the friction it found the opportunities that we might be missing like in one run it might identify a few people or ask about how to export files then it will create one signal about export file to hidden as md file inside this md file it will log which user experience that and every time when it saw this issue happen it will just log and add to the same file system. Similarly, we also have this SEO loop that is running. Every day 9 a.m., it will just go pull all data and research about topics, then publish relevant SEO pages. But doing this data analysis, it might find interesting insights, like one page is actually getting a lot of clicks, but there's not enough conversion from this funnel. Then it can add the signal as a conversion gap for the specific route. And what we really do is that making sure each agent loop both read and write from those shared folder systems. So therefore, we'll put a loop where before it would just watch some post hoc sessions and analytics to prioritize and ideate some growth experiment. But because we have this shared file system, it will analyze data but also look at what are all the other signals that has been identified from other different loops or departments so that it can prioritize and fix a bug that has been reported a few times or grab opportunity that marketing or SEO team is optimizing. Similarly, if we add a loop, is finding a certain keyword is actually has pretty good click-through rate but we don't have organic content around it this signal information can also feed back to the seo loop so you'll be aware of this situation and prioritize some organic content for these specific keywords all those different loops are happening every hour or every day share the same brain and this is where this component effect really taking off and people are also writing this on the tweets where the logging system can actually be the agent which is something similar to what i see here But how do you actually getting started with those crazy compounding loops for your own business? So there are four core components or ingredients that is needed. One is that you need to set up triggers. As I mentioned, it can be multiple different types of triggers. And second one, which is the most important one, is design of the file structure. I do have some best practices that I will take you through. You do need to give agent different tools and connectors so you can do meaningful work. First, it's actually the most important one that a lot of people miss. You want to make sure your code base or environment is set up in a way that allows those parallel and autonomous work happening where many different agents can work at the same time and each verify its own work. I will quickly talk you through each one of the setup. Firstly, how do you set up your code base harness so the agent can have the right environment to do work autonomously? The core point here is that you want to make sure your code base is actually legible, which means agent can easily understand where to change what. And it should also be executable so the agent can easily spin up the dev server locally, as well as the right tooling to verify its work. So first, it is a legible codebase. It's actually not that complicated. OpenAI keeps their agent's .md file to be indexed at roughly 100 lines, which point to all sorts of other documentation systems that they have for agents to progressively discover information. And these two things, I believe a lot of you already doing that. But there's also one part I think is actually very useful is that you can set up a custom link because you can't really rely on an agent to find the random information. for all sorts of different tasks, but you can actually inject those rules into programmatic link chat. So every time when agent is not doing the things right, the warning will be automatically surfaced. Like in our case, we have a pretty complicated mode repo and we don't want agent to use certain repo. So every time when agent write a file, the import from those Lexi folders, it will just surface those type of errors. And there's probably a list of custom links that you can bake in to your code base. But the core idea here is that you want to do the context engineering for your agent. so it didn't always rely on it to find the random information to do the task and second one is that you want to making sure your codebase is executable which means your agent should start the work with just dev server up running ideally cost no token or cognitive load for it so you can actually focus on the work well in our codebase where there's dev local script that's written so the agent can just run the script to get the whole dev server up running And meanwhile, you also want to make sure your code base is actually work tree friendly so that when there are five different parallel agents, it's all working on its work tree. You can still spin up the dev server and test it without conflicting with each other. And ideally, you can also set up some useful scripts to allow agent to jump to a specific state like auth state or unauth state to test specific scenarios. And all those things is providing shortcuts for agent to verify its work easier. And the last one is verifiable. You basically want to give agent the right tools to actually test and log the result. And the one I found best is this PlayRide CRI. It not only allows agent to effectively use the browser, but also it can record a video clips that can be uploaded and attached to the GitHub PR. So it's very easy for you to review whether things are working or not. Alongside a few end-to-end tests for critical flows that we actually care about and want to make sure never breaks, like upgrade flow, signup flow, or in Superdesign's case, create design effectively. And we also provide this PR skill, which defines a list of steps that agent have to do before it can submit a PR. And one of the important things here is that don't get agent to self-verify its own work. It just generally doesn't work that well. That's why in our PR skill, we always tell the agent to spawn a read-only verifier agent with the details back. And if you're interested, I've created a skill called Setup Codebase Harness, capsulating the critical setup that I have done for my own codebase. So you can just give to your own cloud code or codex, then go set up those useful scripts and docs and skills, so your codebase will be much more agentic friendly. I have put the link in the description below so you can try it out for free. If you want even deeper dive, I have walked through the whole setup from scratch step by step in AI Builder Club Workshop, so you can click and follow as well. So this first thing that you should really do, making sure your code base is in a state that agent can self-verify its work. This is going to be helpful, even though you don't write loops. The second part that I'll just quickly talk you through is some best practice I found regarding the file and login system. And there's three main type of files that I thought is good abstraction level. One is artifacts. Those are the outputs of each agent work or findings. This is like shared knowledge layer. And there can be many different artifact types, like docs, the signals. the tasks or many other type of artifacts that is relevant to the loops. Like if I'm running ads, it might even have campaign as a artifact we can use to log and track the campaign performance. In my specific setup, we have an SEO loop, a support loop, a product growth loop that is running on the pricing itself, as well as red loop. So for each artifact, I have its own artifact folder. And in the artifact folder, I would define a readme where it will clearly explain what goes into it, what does not go into it, what's the process for adding a new item in the schema for this artifact. Then for each red command draft, it just has this metadata of font matter as well as the main body about the content itself and also a timeline to log any change that we did for this artifact. Same logic can be applied for like a signals, which in our case is like a part of feedback, ideas, anything that any loop observed. They can create a signal and link to different sources. detailing the role customer feedback, or even support ticket artifact. And the purpose of those artifacts is that those became the shared library or shared files that any loop can just read and write words. And we can even build some small mini apps like this one I built to track all the different artifacts. So I can very easily view, let's say the product signal it ever found and each one can link to another artifact that you created. But it also became very easy for the humans in the loop experience. I can just keep track about what are things that actually needs my attention. But the core thing is here is that you can define those different artifact types and folders that is shared across different loops. And then for each loop itself, I would normally define a contract, which including things like what's the goal of this loop, what kind of workflow it should follow, as well as a backlog list. So the next loop, you can pick up the most important one or update and reprioritize tasks, as well as a list of timelines so you can remember what I did before. In my case, since I have just a few different loops, I have multiple different loop folder. In each loop, I just have this simple readme file. that it serves the purpose of this contract. It will talk about goal and the workflow, then a list of timeline to log what happened to the specific domain. So every time when the loop triggered, it will just read this contract, understand the goal, the workflow, and what happened before. And based on those information, takes math back action. And this contract is extremely useful. And third one is logs. So you might be confusing, like there are already timeline in the loop contract and also the artifact. Why do we still need logs? So the reason we need logs is that I find my day is always a mix of those kinds of review, the output from the loop, as well as executing some of the real difficult or creative work with the agent in those kinds of copilot states. And I want easy way for agent to firstly understand. across different domain what was happening, as well as capture those ad hoc information. And this is where we have this kind of global workload MD file. Each agent, when they finish a big bulk work, it will just write to this file. And also before they start the work, they will also read the last five or 10 entries. So those pretty much are the core ingredients. Let's just set up one together. Let's say this is easy one to start the support loop. We want agent every 30 minutes. It can pull all the recent support tickets we have, draft a response or reply directly to the customers where it has random information. It also logs all the frictions and ideas it identified. First thing is that we need to create some skills. So here I already pre-create a few skills that I know a support person needs to access. Once Intercom skills to fetch all tickets and also Stripe so you can check the payment subscription data as well as Superbase success. So you can debug payment information and render skills to actually fetch the backend logs. There are some scripts that are written as well to make it easier for agent pool information. As well as skill for triage support ticket. So this is part that you will kind of customize based on your own business. In my specific case, I wanted to practically solve problems by defining workflow where you will fetch all tickets that has updates for past X hours. and then investigate the issues the user mentioned and i also wanted to create a artifact of tickets to log the tickets they ever handled and even create engineer tickets logs feedback ideas in the end it should lock what it did so those are skills and meanwhile you generally also want to create a cloud.md file so the agent has a good understanding of your business you can just prompt cloud code or codex to research about your business and save the information to cloud.md file then it will likely create cloud.md file like this that including all the business contacts and also something i generally include is the rules for spawning agent for engineer work and here is where i will explain the reports we have and ask it to get work tree every time it also puts on contract about how it should manage get work tree And those are things that actually apply across any kind of loops. So I'm including the cloud.md file. Meanwhile, I also have this architecture.md, which will be referred inside the cloud.md file. So this is like a general instruction based on the structure I defined. So including instruction that agents should define different artifact types and also define loop domains as well as a convention for logs. So normally I can just point to this file, say, help me set up relevant artifacts, which in my case would be like signal, task, ticket, and doc plus a domain. And the agent should be able to just like read this architecture file and start folding the thing. Then we'll set up different artifact types, like one for docs and one for signal, which covers things like feedback, ID, observation, as well as tickets. This is also a log.md file, which at the beginning is empty. And then I create this kind of domain slash support folder. As I mentioned, for each loop, I found it really useful if we define a contract that is capturing the goal of the loop, the overall workflow and boundaries. and outstanding task list and timeline. Common workflow, I will normally manually run once with the agent, like the test run. Once the test run kind of finished, I will ask it to create a contract, then set up loops. So as an example, I want you to handle my support, which means fetch support tickets from past hour, do-sign analysis, review and draft response saving tickets, and also save signal for product ideas, user frictions, and for clear bugs, you can just spawn agent to fix directly and create a task for engineer tickets. We run this every hour, but let's do a test run first. Then the agent will start doing the work. You can see it already handled a few tickets and each one it created artifacts about the results and also identify some potential customer feedback or frictions that would be useful for prioritized as a product and also log some clear engineer bugs. And this is kind of a process where you can just kind of calibrate with agent to understand whether the workflow is what you want or not. And once this looks right, this is where I will set up a loop. Say, now help me set up a loop, but create a Raimi first as a contract, including the goal, workflow, timeline. in this folder and then set up the loop to the session this should create a proper readme file like this it also set up the loops that will be triggering this session every hour so i actually have created this ripple template called loop engineer setup that's calculating some of the best practice learnings of all the loops that my team has been set up so hopefully you can just copy this folder and instruct it to set up artifacts and spin up new loops i put the link in the description below so you can use this for free as well But if you're interested, we have hours of step-by-step showing me how do I set up those loops from scratch in real alongside 10 hours more deep dive on building agents and production AI coding. So if you want to set up your first loop for your business, definitely come and join. I hope you enjoyed this video. Thank you and I'll see you next time.

Get free build your first AI agent 101 course: https://clickhubspot.com/b093f6

🔗 Links
- Github repo for codebase harness + loop engineer knowledge template: https://github.com/JayZeeDesign/loop-engineer-template
- Step-by-step workshop in AI builder club: https://www.aibuilderclub.com/lp/loop-engineer
- Follow me on twitter: https://twitter.com/jasonzhou1993
- Try Superdesign: http://superdesign.dev/