wtf is Harness Engineer & why is it important

Thanks to HubSpot for sponsoring this video. So something really big actually happened in December 2025. And most of the people didn't even realize that. Andrew Cupsie tweeted about this last week. It's very hard to communicate how much programming has changed due to AI in the last two months, specifically since last December. And Greg from OpenAI also talked about this. Since December, there's step function improvements in what model and tools are capable of. And a few engineers have told him that their job has fundamentally changed since December 2025. So what actually happened in December 2025? In short words, the latest model introduced then is finally ready for fully autonomous long-running tasks. So with AI, the ultimate dream is always that while we are sleeping, AI can just work on tasks fully autonomous day 24-7. Even back 2023, the most popular project, if you remember, is called AutoGBT. It is first time those fully autonomous agentic systems was introduced. And they have fairly basic and simple architecture that using GPT-4 as a model to automatically break down a list of tasks based on user's goal and has simple memory storage to store the result. And people were doing some pretty crazy stuff like just give a go, make $100,000 and let it loop through tasks infinitely until completed. Back then, the system just break and fail miserably because the model is simply not ready. But since December last year, this really changed. The models have significantly higher quality, long-term coherence, and they can pass through much larger and longer tasks. And we saw all sorts of different experimentation came out from the industry. Firstly, from January, we got this super hot concept called rough loop. And most basic and simple agent iteration loop to force model work longer so that it can take more complex tasks. We just follow the model with some simple condition checks. But already, we start seeing the difference. One week later, Cursor also released their experimentation. where they use GPT 5.2 to automatically build a browser from scratch with 3 million lines of code. And Entropic also released this experimentation they had where they get a team of cloud codes to automatically working on a C compiler from scratch for two weeks. And in the end, it delivered a functional version with zero manual coding. It can even run Doom inside this compiler as well. At the same time, OpenClaw started getting attention and had this explosive growth that we've never seen before. It was very difficult to understand what was going on with OpenClaw. Because from outside, it's very easy to categorize OpenClaw just be another menace, but live inside your own computer and can also access from Telegram. Like, why is it so popular? and only later after i use the Dplay i realized that the real difference is that open claw represent this type of always on long-running for atomic agents that is very different from all the other agentic system we used before where human is the main driver to prompt for the next action OpenClaw is always on and it is proactive and this autonomous feeding is created by a fairly simple architecture where it has memory context layer with a trigger and a cron job to automatically take actions and have the full computer access which is powerful environment it can operate in and I believe OpenClaw is the first project that really opened up the biggest paradigm shift in 2026 that we are moving from a co-pilot simple task-based agent system to those long-running fully autonomous agent something that's always on always ready, autonomously delivering super complex coordinated work. This is a critical shift you have to understand. The model today is actually much more powerful than you think as long as you design the right system to unlock it. And this is the crux of what I want to talk about today. The harness engineer to enable long-running autonomous systems. If it's the first time you hear about harness engineer, this is like evolution from what we've been previously talking about which is context engineer or prompt engineer. So previously we really focused on how do you optimize the prompts within the effective context window to get a model have the best performance for a single agent loop session. But Hardness Engineer is really focused on those long running tasks, which means how do you design a system that can work across different sessions and multiple different agents? And how do you design the right workflow to making sure the relevant context will be retrieved for each session and right set of toolings to attract most other models? This is a fairly new concept. But the good thing is that industry already converged on some best practice that you can use from Antropic, Vercel, Lanshain and many others. More goes through each one of them one by one so you can see the patterns. But before we dive into this, with this paradigm shift fully autonomous agents, one of the biggest opportunities for the next 6 to 12 months is build an open clock for a certain vertical, which means you deeply investigate and understand the end-to-end workflow of a certain vertical and build an autonomous agent with the correct environment and tooling to enable the end-to-end process. That's why I want to introduce you to this awesome research HubSpot did on the AI adoption in email marketing report. It is a fascinating report for you to understand for a vertical like email marketing where people actually use AI today and what are the gaps. Because this report showcases clear workflow and opportunity in email marketing that you can potentially automate. They surveyed hundreds of email marketers from top companies to understand exactly how AI is reshaping their workflows. They talk about why marketers are still doing a lot of heavy editing, what were the cause to it, as well as the biggest challenge they are facing today when implementing AI in the email marketing. And each of them is a big opportunity for you to build 40 autonomous agents. They even dive into the specific KPI that they care more about and AI has shown proven results, as well as what exactly things email marketers really want from AI. So if you're a builder who are thinking about the next bid agent product to build, I highly recommend you go check out this awesome resource. I have put the link in the description below for you to download for free. And thanks HubSpot for sponsoring this video. Now let's get back to Harness Engineer for long-running agent systems. And at high level, there are three learnings I took away from those. One is that for long-running task agents, the critical part of system design is creating this legible environment where Each sub-agent or sessions can actually understand where things are at. Most likely there are some workflows that can be done to enforce the legibility of the environment. And I'll expand a bit more on that. The second is verification is critical. You can improve system output significantly by allowing it to verify its work effectively with faster feedback loop. And third is that we need to trust the model more. Instead of building specialized tooling, they wrap a lot of reasoning and logic prematurely. We should give model max contacts with generic tooling that they natively understand and let it just explore like human. And I'll unpack those three things one by one as we go through each block here. Firstly, this is Antropix effective harness for long-running agent's blocks. So they've experimented using Cloud Code SDK to build a specialized agent for super long-running tasks like build a clone of Cloud.ai website. The very first failures they observe is that firstly, agents tend to do too much at once. Essentially, it will always try to one-shot the whole app. And this led to the model running out of context in the middle of its implementation and leaving the next session to start with the feature half implemented or documented. Then the agent would have to guess what actually happened and spend substantial time trying to get the basic app working again. And the second failure they observe is that agents tend to declare job complete prematurely. You probably experienced this a few times yourself as well. The Cloud Code or Cursor will just claim the project or feature is completed. But once you test it, it actually didn't work. So their approach to solve those default model failure behavior is that first they set up an initial environment that lays the foundation for all the features that are given prompt requires, which set up the agent to work step by step and feature by feature. So this is kind of similar to the plan or PRD approach that we normally took. The second is that they start prompt each agent to make incremental progress towards its goal while also leaving the environment in a clean state at the end of each session. What they did is starting to design this two-part solution. First, they will have this initializer agent that uses a specialized prompt to ask model to set up an initial environment with the init.sh script, which will set up dev server, for example, so that next model don't need to worry about those things. And also a claw progress.txt file that keeps logs on what agents have done, as well as initial git commit that shows what file has been added. Then a coding agent for each subsequent session to ask the model to make incremental progress, then leave structured updates. And all those efforts are really try to serve one purpose is how can they define an environment where agents can quickly understand the state of work when starting with a fresh contact window. So the workflow is that the initializer agent would firstly try to set up a environment or you can call it documentation system to track and maintain the overall plan. And the environment they design here is firstly, they will have a feature list documents to prevent agent one-shotting the whole app. or prematurely considering the project complete. Instead, they will get the initializer agent to break down the project into over 200 features. Log them in a local JSON file looks something like this, where each task has detailed spec as well as pass or fail state. At default, all tasks will be marked as fail. So the force model to always look at the overall project goal and the progress, pick up the highest priority task and do the next thing. But to make this workflow work, they also need a way to force the model, leave the environment in a clean state. after making the code change. In their experiments, they found the best way is to ask a model to commit its progress to Git with descriptive comment message and write a summary of its progress in progress file. But with just documentation and contact environment itself, it's not enough because model at default has this tendency to mark something as completed without proper testing. And in the beginning, they were just prompting Cloud Code to always do the tests after the code change. by doing unit tests or API tests for the dev server. But all those things will often fail to recognize that a feature is not working end-to-end. But things really start changing when they give model proper tooling to do the end-to-end test by itself, like Puppeteer MCP or Chrome DevTool, where Adrian was able to identify a fixed bug that were not directly obvious from the code itself. So basically, they are setting up this structure where they have the initialized agent to break down the user's goal into a list of features alongside init.sh to be able to run the dev server and progress files. So the next coding agent can just read the feature list to get an understanding about the overall project plan and pick up the high priority task and progress file and git log to understand where things are at. Then run india.sh to start dev server immediately and do end-to-end tests to verify the environment is clean so that it can get a full picture, faster feedback loop while each new session and context window happens. In OpenAI's blog, they talk about very similar things. You have to make sure your application environment is legible. They make the whole repository knowledge, the system or record. Initially, they put a gigantic agents.md file and it fell in predictable ways because it's just too much context for any agent to manage and maintain. So what they did is design a proper dog environment structure and treat the agents.md file as a table of contents. So they set up this documentation system from architectures, the design docs, the execution plan, DB schema, part of specs, and design front end plan, security, and many more. And put this table of content into agents.md file. So the agent can actually retrieve back relevant information when needed. And this enables progressive disclosure. And OpenAI actually do it even further. They will try to push not only the code knowledge, but also Google docs, Slack message, all those other fragmented information. feed the data into the repository as a repository local version artifacts. So the agent can also retrieve. Because from the agent's point of view, if anything can't be accessed in the environment, then effectively it doesn't exist. But again, documentation itself doesn't really keep a fully agent-generated codebase coherent. They also introduce certain programmatic workflow to enforce invariants. For example, they layer domain architecture with explicit cross-cutting boundaries. which allows them to enforce those rules with custom checks, linters, and structural tests, which can be automatically triggered and injected by every Git pre-commit. And those type of architecture usually you will postpone until you have hundreds of engineers in traditional software company. But with coding agent is an early prerequisite. Within those boundaries, you allow teams and agents to significant freedom in how solutions are expressed without micromanaging and worrying the architecture is going to drift. Meanwhile, they also improve the code base a lot. For example, they made the app bootable per Git worktree, so Codex can just launch and drive many different instances. And they also wired Chrome DevTool protocol into the agent runtime so that the agent can reproduce bugs, valid fix, by DOM snapshots, screenshots, and navigation. And with all environment and workflow setup, their repository finally crossed a minimal threshold where codecs can end-to-end drive a new feature. So every time when codecs receive a single prompt, the agent will start to validate the current state of the codebase, reproduce a reported bug, record a video demonstrating the failure, implement a fix, validate the fix by driving the application, record a second video demonstrating the resolution, and eventually merge the change. So those two blocks struck his very good learning and necessary harness system you need to put in place for a fully autonomous system. There are also third learnings. Quite often when built agents, especially vertical specific agents, our tendency is to build specialized tooling to do domain specific tasks. The learning goal is that large learning models almost always work better with generic tools that they natively understand. Russell released this awesome article about how they redesigned their task to SQL agent. So they spent months building a sophisticated internal task to SQL agent D0, which specialized to heavy prompt engineering and careful context management. But as many of us experienced before, those type of systems kind of work, but it's very fragile, slow, and require constant maintenance. Because every new edge case has happened, you will need to inject new prompt to the agent. But later they tried one thing that totally changed the trajectory. They deleted most of the specialized tool from the agent down to a single batch command tool. And with this much simpler architecture, the agent actually performed 3.5 times faster with 37% fewer tokens and success rate increased from 80% to 100%. Similar learning has been shared from Entropy team as well, where they talk about instead of having specialized search, linked, executed tools, they just have one batch tool where you can run GripTel, NPM, NPM run, linked. And fundamentally I think it's because all those large language model is much more familiar with those code native tools that has billions of training tokens versus bespoke tool calling JSON that it needs to generate. And I've talked about this in programmatic tool calling video that I released last week. And I believe it is similar fundamental principles here. But the foundation of those simple architecture is again the good context and documentation environment where model can use generic tools to retrieve context progressively. And in the same case with OpenClaw, one reason OpenClaw is so interesting is that they have a surprisingly simple but effective context environment. They have a list of documentation to store core information. With this foundation, they only have the most basic tooling like read, write, edit files, run bash commands, and send messages. All the rest is coming from giving agent environment to retrieve relevant context, plus a big scale libraries to expand capabilities. So those are three practical learnings about how to do hardness engineering for long-running complex agents by setting up a legible context environment to enable each session to grab context effectively and write workflow and tooling so that model can verify its work effectively, drive faster feedback loop, and trust agents with generic tools that it natively understands. If you're interested, I'm going to share more in-depth about how do I take these learnings and transform into a development lifecycle process. In AI Builder Club, we have courses and work through about Vibe coding and building production agents. And every week, myself and industry experts share the latest practical learnings. So if you're interested in learning what I'm learning every day, you can click on the link below to join the community. I hope you enjoyed this video. Thank you, and I'll see you next time.

Get free AI Adoption in Email Marketing Report: https://clickhubspot.com/cb84e9

🔗 Links
- Join AI Builder Club: https://www.aibuilderclub.com/
- Try Superdesign: http://superdesign.dev/
- Follow me on twitter: https://twitter.com/jasonzhou1993