What is an Agent Harness? and How to build a great one!

Everybody talks about agent harnesses but what exactly is a harness? Now even people who are actively building agents can't always give you a clean answer. The word gets thrown around constantly but nobody really agrees on what exactly it means. So in this video I want to do three things. First define what a harness actually is and just as more importantly what it's not. Then We will walk through nine components that I think makes a modern harness. And finally, we'll build a tiny one in Python so you can see exactly what is going inside. This is going to be especially important for people who are thinking about building agents and harnesses. In simple term, a harness is a fixed architecture that turns a model into an agent. So if you think about modern LLMs or models, These are just one-shot text generators. You ask a question, it answers and stops. A harness is what gives it the ability to take action, see the consequences, and keep going until the problem is actually solved. So think of a model as the engine and the harness around it as the car. That's what makes an agent. So a really good example of this is... agentic coding tools like codex cursor clock code windsurf these are all harnesses each one started from a concrete problem making a model write and edit code across a real repository and i think they have a conversion remarkably similar architectures Now we're going to look at that architecture in a minute, but first I want to talk about something else that you probably have heard about, and this is frameworks. So think about things like LangChain, LangGraph, Autogen, CrewAI. These are not harnesses. And I think this distinction is really worth making because right now people are using these terms interchangeably and it's kind of getting confusing. So a framework gives you abstraction. think about state graphs, chains, memory contents, and retrievers, you as a user have to wire them together. The fundamental assumption is that you, the human architect will configure these pieces together. Now harness on the other side is from opposite direction. There's no assembly step. It basically ships a working agent and in simple terms, it's just a while loop with a tool registry and permission layer and everything comes wired together now another way to think about this is that a framework is built for a human to assemble an agent a harness is built for the agent itself to a task and in big picture you just provide the goal the harness will handle the rest so in the rest of the video we're going to primarily focus on harnesses and what make them interesting okay so what exactly is inside a harness i would say there are nine main components that you need to consider if you are building an agentic harness now we're going to go through the list this is mostly opinionated architecture but something that i have seen to work really nicely in practice i'll try to tie it together to cloud code because i think this is an example of a really great harness put together Okay, so the first component is the while loop. This is basically the foundation. It's the outer iteration loop. The harness is at its core of while loop. The model reads its system prompt decides which tool to call, runs the tool, feeds the result back into the context and loops again. And this process keeps the model produces a text only response. or it uh hits a maximum iteration cap now we're talking about uh text only models but the same can apply to multimodal models as well so think of this outer loop as the whole engine that runs everything now number two is context management on every turn the tree grows as you encounter more user messages more tool calls we're going to see that you hit the context limit of your large language model. So the harness has to decide what to keep verbatim, what to summarize and what to throw away. In Cloud Code's example, the budget used to be around 200,000 tokens. Now they have increased it to 1 million token in case of Opus. But let's say when you are reaching almost half of it. or maybe 80 to 90 percent it triggers a compaction some of the most recent messages are going to stay in full everything older gets summarized now this compaction is very important and it can have some real bad consequences if not properly did so you need to be very careful about context management now the third component is skills and tools So tools are the primitives that reads a file, edit a file, run bash, search code. Skills are a top, a layer on the top. So they are how organizational knowledge gets encoded. Usually you're going to see them in my own files. Now to think about tools and skills, I would say tools are universal. Skills are specific to your team, your workflow. And then there is the registry. So it tells what is available, what permission each thing needs, how the call gets dispatched. Now, the fourth component is subagent management. Now, at some point, a task gets too big or too parallel for a single conversation thread. So the harness is going to create subagents that work in isolation. Each subagent gets its own session. its own restricted set of tools in a focus system prompt that says you're working on this specific task. Now, the idea over there is to span, restrict and collect the outputs. That's kind of the pattern you want to use. If you want to learn about generative AI in a structured way, you will love today's sponsor, which is Coursera. They have a number of different courses on generative AI. Here are the four that I'll highly recommend for you as a beginner. Start with Google AI Essential. It's about 10 hours taught by Google's own AI team. Over 1.7 million people have already enrolled. Next, Wonder Builds Prompt Engineering for ChatGPT. This is where you learn the actual patterns, chain of thought, few short, persona prompting, etc. If you are a developer, jump to their generative AI with large language models built with AWS. It's a hands-on lab. You fine-tune, deploy, and ship real LLM applications. And if you want a full career on-ramp, IBM's AI Developer Professional Certificate is for you. You get to build chatbots, apps, rack, and agents. Right now, get 40% off. three months of coursera plus through the link in the description now back to the video number five is built-in skill so we already talked about skills that you provide as a user but every harness ships with a baseline set of kit that are going to work out of the box so think about file operation read write edit search or cell execution code navigation things like that now for modern harnesses these are really non-negotiable if your agent cannot read or edit files it isn't a coding agent so beyond the primitives modern harnesses also ship with higher level skills for example a harness can have a skill of how to make a git commit how to open a pull request how to run test results now some of these built-in skills are going to be specific to the vendor or creator of the harness. Number six is session persistence or memory. So a long agent session is stateful. If the process crashes, you lose everything unless the harness writes state to disk. And the way the modern harnesses do this is pretty elegant. So typically they're going to use append only JSON files or maybe markdown files. So every message, every tool results, every compaction event gets one line. Now, the beauty of this is that you can resume exactly where you left off. I actually recently attended a talk from the Anthropic team where they were discussing about how they built managed agents. I'm going to cover that in another video, but they had the session management separate from the harness itself which was i think a very interesting design number seven is system prompt assembly now this is the one that will surprise most people the system prompt is not a static string it's basically a pipeline that walks ancestor directories looking for specific types of instructions so if you have cloud.md or agents.md It's going to inject those into the system prompt. Now, you also want to be a little careful here because most of these third party harnesses or even the first party harnesses have really strict record prompt caching. If you dynamically introduce components to the system prompt, that is going to break the caching, right? So you need to be careful about that. But in certain situations, you want to agree. uh assemble the system prompts okay number eight is going to be life cycle hooks so this is extensibility scene hooks let you inject custom logic before or after a tool runs without touching the harness itself so a pre tool hook files before execution it receives the tool name the input and can allow deny or modify the call a post tool hook runs after and can inspect the results so the protocol is kind of stirred think about a json file with exit codes for allow or deny now the beauty is that hooks also enable uh intercommunication between different heart and hooks are how enterprises today adopt harnesses themselves now let's talk about number nine permissions and safety so this is the layer that makes the difference between a useful tool and a dangerous one modern harnesses define a hierarchy of permission modes you can have read on look space right full access each tool declares the minimum permission it requires now the job of the harness is to enforce that at dispatch time before the tool even ever runs and for tools like bash the harness even classifies the commas dynamically So let's say if you say list files is going to be read only. If you want to delete something that will need full access. And the harness figures it out by passing the command string. On top of the static permission, you get interactive approvals. The agent can pause and ask, should I run this, right? So before executing anything dangerous, you want to have this safety layer built into the harness. All right, so these are the nine components that I think every harness needs to have. Iteration loop, context management, skills and tools, sub-agents, built-in skills, session persistence or memory, system prompt assembly, lifecycle hooks, and permissions. Now, the easiest way to actually understand a harness is to build one. So let's write a minimal version of Python. Nothing fancy. just enough to see all these components together. Okay so in this part we're going to quickly go over reference implementation. So think of this as a structure or template that you want to use if you're building a harness. Okay so the main engine is the while loop that basically controls everything. It assembles the system prompt and starts looping. Now on every iteration the context is going to be compacted if it grows too large all of these things plus the two calls and calling to sub-agents are going to be implemented within this while loop you also want to cap how many iterations are going to be in this loop so that it never runs forever this is really the entire engine every other file in the project exists to support these few lines now this code implements simple context management in this very simple form we are just doing compaction so if the history grows beyond a certain point we just summarize some of the older conversations and put them together now there are more advanced compaction techniques but this is a very simple reference implementation now if you're making two calls you also need to decide whether you're going to bring in everything that is done within the tool call or only the input and outputs so those are the design design decisions that you'll have to take as an architect okay this code implements a simple tool and skills registry so every tool in the harness is described by a small data class a name what type of permissions that are going to have in a handler function and one live line description the registry is just a dictionary that maps the tool name to that record now there are a few functions calling register adds a new tool calling get retrieves one for dispatch calling descriptors return a lightweight version of the list which is going to contain name permissions description that we are going to send to the model so it knows what is available scales are registered the exact same way they are just tools whose handle handler reads a markdown file at invocation time next is a sub-agents uh so you can implement multiple different sub-agents this code looks at three different things for exploration general and then verification now each archetype has its own permission levels it's on a restricted tool list and its own focus system prompt Now every harness also needs to have built-in primitives. These are the non-negotiable tools every coding harness must ship with. Example of this would be reading files, running bash commands. Now this also depends on the type of work you want your agent to perform. Now in this case one thing to keep in mind these primitives need to use pure standard libraries. You don't want to rely on framework dependencies. uh which is going to be critical because that actually enables the model to take actions okay so here is a simple reference implementation for um session memory or context persistence now every event the agent generates gets written to disk as one line of json you can also use markdowns but json seems to be the better choice the append method opens the file in append mode writes the event and immediately flashes it that way if the process crashes after the next line this one is already safe on disk the replay method reads the file back line by line and reconstructs the full session because the file is append only two runs of the harness can share the same log without stepping on each other if the harness dies the file does not this is the whole durability story if you want to persist memory okay next one is system prompt assembly so you don't have to have a fixed system prompt you can actually dynamically load things into the system prompt so for example you can load agents.md cloud.md or any other memory files that you have stored into the system from dynamically just by reading a directory and reading files from disk one thing to be aware of the order matters here so keep the static part first and then dynamically load content a second otherwise you're going to break the prefix caching okay so next one is hooks which are used for extensibility now there are two different types of hooks one is pre-tool hook and the other one is post tool hook and the idea is the pre-tool hook fires before any tool runs and can either allow or deny the call a post tool hook fires after the tool runs and sees the output it cannot block anything it's there to audit can be used for logging and observability okay the last component is permissions and safety so each tool declares the minimum permissions it needs it can be read workspace or full and now the harness needs to provide that extensibility and control the permissions of every tool Now there's one more thing that you need to be aware of. The same tool can be safe or dangerous depending on the command. So we classify it dynamically. Safe commands like list, concatenation and grip stay at read only. Dangerous commands like delete, sudo or shut down jump straight to the full access. So anything else gets workspace level. on top of these static rules the agent can also pause and ask the user for explicit approval before running anything destructive and this is the part that you need to implement within your hardness now these are the components that i think every harness needs to have now do let me know your thoughts whatever components your harness have and if you're interested in technical contents like this make sure to subscribe to the channel anyways thanks for watching and i'll see you in the next one

To apply 40% off 3 months of Coursera plus - https://imp.i384100.net/c/7245724/3880401/14726
Google AI Essentials - https://imp.i384100.net/1GW56D
Prompt Engineering for ChatGPT - https://imp.i384100.net/gRWb9g
Gen AI with LLMs - https://imp.i384100.net/n421aV
IBM AI Developer - https://imp.i384100.net/R06yzX

In this video, I define what an agent harness is (and what it isn’t), explain how it differs from frameworks like LangChain, LangGraph, AutoGen, and CrewAI, and show how a harness turns a one-shot model into an agent that can act, observe results, and iterate toward a solution. I break down nine key harness components: the while-loop engine, context management and compaction, tools vs. skills with a registry, subagent management, built-in skills, session persistence/memory, dynamic system prompt assembly from files like CLAUDE.md or AGENTS.md, lifecycle hooks (pre/post tool), and permissions/safety with dynamic command classification and user approvals. I then walk through a minimal Python reference implementation covering these pieces, including tool descriptors, subagent archetypes, append-only JSON event logs, and prompt assembly considerations like prefix caching.


My voice to text App: whryte.com
Website: https://engineerprompt.ai/
RAG Beyond Basics Course:
https://prompt-s-site.thinkific.com/courses/rag
Signup for Newsletter, localgpt:
https://tally.so/r/3y9bb0

Let's Connect: 
🦾 Discord: https://discord.com/invite/t4eYQRUcXB
☕ Buy me a Coffee: https://ko-fi.com/promptengineering
|🔴 Patreon: https://www.patreon.com/PromptEngineering
💼Consulting: https://calendly.com/engineerprompt/consulting-call
📧 Business Contact: engineerprompt@gmail.com
Become Member: http://tinyurl.com/y5h28s6h

💻 Pre-configured localGPT VM: https://bit.ly/localGPT (use Code: PromptEngineering for 50% off).  

Signup for Newsletter, localgpt:
https://tally.so/r/3y9bb0


00:00 What Is a Harness
01:50 Harness vs Frameworks
03:19 Nine Core Components
03:49 Loop and Context Control
05:41 Tools Skills and Subagents
06:57 Sponsor - Coursera
09:03 Memory Prompts and Hooks
11:41 Permissions and Safety Layer
13:04 Build a Mini Harness
13:18 Reference Implementation Walkthrough