You Can Learn Hermes Agent Harness In 20 Min | Loop Engineering, Memory System, Self-Improving AI

Hey everyone, this is Sean. So today we're going to walk through Hermes Agent Harness and its loop engineering system. This is one of the most popular Harness Agent system right now, and it's an open source project. If you check the GitHub, they've got more than 200,000 stars. in a very short period of time so it's basically a self-improving ai agent built by this research lab called new research and it's a built-in learning loop it creates its own skills from experience instead of you telling the ai like clock code to make skills and then it also improves over time when you keep using it because it persistently stores knowledge in your local machine like your macbook but anyways this is all the buzzwords we're going to jump into this Hermes agent system design like usual we're going to test it as well using their desktop app give you a quick example it's like a clock code right you can see that it was thinking and then it answered me in like a pikachu style it said pikapi just hang out in my directory because it's literally a bot living in my local machine files whatever building fixing today and also you can interact with it on whatsapp and then just say good morning It's a little bit slow. I'm using a Gemini 3.1 model for this. It took like a good five seconds. And now it's basically just using WhatsApp as the gateway to receive information from my WhatsApp account. So I can just text it and then ask you to do things for me. And say, pick up a good morning. You probably realize that I've changed this personality a little bit. And there are more interesting things I can show you. For those who have watched my previous Asian Harness Lube Engineering and Memory System, you might realize that this chart is somewhat similar. This is what the previous chart looks like. And now if you look at Harness for Hermes Asian, again i added a bit more information especially the difference between the hermit's harness versus a generic harness system, I marked the differences in red. But we're going to walk through this step by step. Also, I prepared some testing examples to cover harness, memory, skills, loop, gateway, and LMOPS or eval. Because a lot of people were asking me if I could implement some real examples. So here we are. Without further ado, let's jump right in. Okay, so firstly, let's look at the foundation of Hermes Harness Agent. It's a harness that runs on your local machine. You can use either local command lines, Docker, SSH, or a virtual VPS. There are two main ways. interact with Hermes agent. One is using your favorite communication apps and in my case I used WhatsApp or you can just use your desktop app with the Hermes agent. So you can come to Hermes official website hermes-agent.newresearch.com and just click on download for macOS or you can click on Windows over here to use the command line to install it. Essentially it's still a chatbot just like claw code and you ask the question to either the communication app or the desktop version. It's very similar to OpenClaw where you can use WhatsApp to control it. So after user sent the prompt, the user prompt with the current chat history and system prompt will be fed into a working memory. And then this LLM agent is basically the Hermes agent that will be answering your question. And it will be using a loop engineering over here to call up some tools that are only on Hermes. And eventually after finish the task, it's going to end the loop. and then send the user a reply. Just literally sending a question throughout this entire ephemeral agent run and then get the user reply. Just like my Hermes was greeting me like a Pikachu. And by the way, how did this happen? I'll show you right now. So the system prompt for Hermes is interesting. It's called soul.md. I really like how they name it. It says soul. If you come to the right hand side, you can see there's a little file bar and this is your local files. And you can go find out there's a dot Hermes folder. And we open that. And if we scroll down, You will see there's a file called soul.md. Double click on that. And you can edit this yourself, right? But since I've already edited it, after you edit anything, you can just click on save. I basically say you talk like Pikachu. Every reply starts at end with pika, pika, pika p. If you are excited, say pika p with the lightning. If you're sad, say quiet pika. I'm pretty sure you can do something similar in Claw Code for this. In Claw Code, something similar will be in customize, general. And you can see there's an instruction for Claude. You can tell Claude to do certain things and not to do certain things. It's basically the system prompt used across the entire agent harness. If you didn't watch my previous video on harness and loop engineering, harness is basically a set of tools that allows you to control over this really powerful horse, which is the LLM itself. In our case, the LLM we're using is Gemini because I've got the Google credits. I don't want to burn my Anthropic credits, but feel free to use Anthropic. This is the agent or the horse. And the harness is this entire thing. It's like a horse harness that you ride on and you make sure that the horse is not going anywhere random and it's using the tools accordingly. to move in the right direction. Harness is a very vague concept. Just don't fantasize it. It's nothing complicated. It's just a buzzword. Now we're going to continue to build up this harness right here for Hermes. But what I'm going to show you is what a loop is. The concept of loop got really popular because these days people build a lot of agents. You don't want to always tell the agent what to do. You want the agent to figure out what kind of problem you should send, what kind of tools you should call. Let's take a look at what tools Hermes agent actually have. to enable this loop engineering we're going to cover this briefly and then i'll show you real examples so first of all the agent of Hermes is having a set of tools that include the terminal the browser okay you can use your laptop's terminal you can start browser it can delegate tasks which means that you can spawn up some sub agents for you for example you can ask it to spawn up an agent which will call your claw code cli to write code for you if the task is basically fixing some bugs on my github and then it can schedule a cron job A ChromeJob is basically, you can schedule something that will happen at a certain time. You don't need human intervention to do it. For ChromeJob, let's just try it right now. In HermesAsian, I can say, help me set up a ChromeJob that will tell me a Pokemon joke every minute in the next 10 minutes. Okay, I'm just going to send this. And also, I'm going to send this to our HermesAsian WhatsApp interface too. Instead, I'll just say, tell me a developer joke every... It just quickly called this a ChromeJob. And it says, it repeats 10 times for the next 10 minutes. Because it says it's hanging in the terminal UI, it won't be able to print it, so I need to ask it. I just ask, what's the first joke? And you can see on WhatsApp it also got scheduled, the Chrome. It says the joke machine is powered and scheduled. I've set up a Chrome job to send you a fresh joke every minute in the next 10 minutes. You should receive the first one in about a minute. We'll see about that, okay. Why did the squirtle across the ocean to get the other tide? Pika pika. Okay, so that was Chrome Job. It can do skill management as well. It can connect to MCPs as well. It depends on the question that the user asks. The agent will just leverage these tools until it's done, and then it will send a reply to you. Okay, the example we just saw was Chrome Job. I'm going to show you some more examples later too. Let's test a few hardness and lube examples. The first one is use a terminal tool to find out what OS version I'm on and read the last 10 lines of my shell history. Let's do that. Start a new session. I'm just going to ask this exact question to it. So you can see it ran this on my local machine, and then it figured out that I'm on Mac OS 26.5. And the last 10 lines I used in shell history is basically text, git status, git status, git status. I always like to check git status. And here what happened was that it ran through this agent run, and then it called the terminal to do the task for me, which is checking. what system I'm on. This is a very simple example, but you can do more crazy things. You can ask it to use the local machine terminal to do things for you. If you're using Cloco, you're very familiar with this already, right? There's nothing special about it. I'm just showing you the example that this is a harness and there's a loop because after I fetched it, it will stop and tell me and reply to me. Let's look at the cron job as well. You can see they sent me two jokes already. Why do Java developers wear glasses? Because they don't see sharp. Pika pika, I would tell you a UDP joke. but I have absolutely no guarantee that you would get it. I'm not going to wait for acknowledgement anyways. Pika. Okay, this one I didn't get it. So one more example. Browse to YouTube channel Sean's AI Stories. Title plus view is my last videos. Create new video titles for this current video I am filming. So here we're asking it to use the browser basically. Okay, you can see that it started doing the searching. All these things already exist on ClockCode. I'm just explaining it from a system design perspective and showing you examples, okay? and then also i will show you what's the difference between hermits and clock code so please bear with me all right so i need to run a python script so you can see also use some clicking tools oh there's an error my website was wrong trying this new one it saved a little memory here okay it said user profile updated this is something i'm going to cover in a bit but spoiler alert Hermes agent will save agent memories by itself. It's basically in your local files too. If you come here within the Hermes folder, if you scroll down in memories, you can see there's a memory.md. If we double click on that, it's stored YouTube here. YouTube scraping quirk. So because it realized that it had a mistake, so it... update itself with the memory this is a self iterating process of this harness good so it read my previous YouTube titles it realized that I've got this video that's got a hundred thousand views and since I'm filming a new video now it gave me some new titles so what happened here was the user which is me who asked the question through the desktop and then it went through this Asian run and then this Asian harness Gemini that I'm using was basically used calling the browser right here to search up my YouTube channel And then it came back and said, hey, I didn't find anything. So it basically stopped. There's an end loop guardrail. The guardrail stopped and said I didn't find anything. So I realized that, okay, the previous URL was wrong. So I input it again. It called the browser again. And it also called the terminal to run some Python script because you need to do some task. And eventually it came back with the answer. There's a mechanism that says, okay, end the loop. No more tool calls. And then send the answer. These are two quick examples of what a hard loop engineering is. You see, you're already using it every single day. It's nothing fancy. So don't get overwhelmed next time when you hear these terms. It's just a buzzword. So what happened just now was this memory.md that we saw regarding YouTube memory. Hermit's agent, after or during the agent run, it's going to start to create its own skill.md and memory.md, depending on if it's feeding into the procedure memory or if it's feeding into the semantic memory. And we should add one more arrow here, basically. So essentially, the way that Hermit's agent works, is that it's storing its own skills and memories completely locally nothing stored on the cloud it has its own self-improving loop every time when they realize that something is a mistake that you should learn from or it's a repeated task it will start to summarize okay here's some new memories i should know for the future in order to serve this user well it started to save these files into these two chunks of memory system first one is called procedure memory It's basically how you act, how the horse agent should be acting. And it's usually saved in this directory called Hermes, Skills, Skill.imd. Hermes, Skills, and any of these. Okay. Autonomous, AI agent, Cloud Code. There's a skill for how Hermes will use Cloud Code here. So it will basically delegate any coding to Cloud Code CLI. Show the exact how you should do it, like install Cloud, run it. and do the authentication and all these kind of things and then what we just saw earlier was memories memory.md is saving the semantic memory which are some durable facts or things related to the user profile so if the user has a habit or has some important facts or information you should remember then it will be saved in the memory.md and what's interesting is that for Hermes instead of doing the embeddings it's actually just using plain text okay it's using the top kick keyword instead of doing the embedding or the rag system it's very interesting because i'm not too sure why but just doing text so remember that we say the agent run is ephemeral now that with working memory to be fed into the agent for all the loop engineering which includes procedure memory semantic memory and later we'll cover episodic memory This will become a more complete memory system that will feed into the agent runs, which is making the harness more elegant. And the third pillar we haven't covered is called episodic memory. It's basically the chat history or some dated events. It can be racked or sequel into the working memory. After every agent run, the information will flow into... episodic memory what's different in Hermes is that the data is not saved in the cloud it's saved in this place called state.db again let's come back here on the right hand sidebar if we again open Hermes folder square down you can see at the end there's a state.db and I can click on preview anyway this is basically a database that will include all those info in your chat history but also over time it will start to consolidate some of those chats using the Hermes auxiliary models which are cheaper non-important models to do this summarization task so that it will distill some facts into what we call the semantic memory or that memory.md. So you can see this harness is running really in a self-improving autonomous situation, which is really cool. I know this chart is getting more complicated than usual, but I feel like this should be a complete summary for how the Hermes agent works. And let's go through a few more examples. Let's test out some of the stuff for our memories. Save to memory that my favorite testing framework is PyTest. So this is an explicit saving, right? It updated the memory file. So if we go to memory, You can see one more block, which is user's favorite testing framework is pytest. That was an explicit ask from the gateway. The tool is basically, let's go update the memory over here. Search our past sessions. What was the very first thing I ever say to you? Now it's using the searching tool in the loop engineering iteration. Say the first thing I say is, what can you do, Hermes? And what makes you special? Let's double check that. That's right. And it failed because I couldn't log in at that time. So what happened was I asked the question through the interface. it went through the prompt and in the working memory because we asked that so it created the information from episodic memory from state.db it returned the memory into the agent the agent was like oh okay this is what you said let's try an example related skills which is procedural memory so we're going to say create a skill called video prep that captures how i format my video scripts spoken english define jargon on inline no m dashes closes with you can build anything you can learn anything this is an explicit call to update the skills it used the skill management tool remember in the loop engineering there's a tool called skill management and this is what it did and say pick up p i have successfully created the video prep skill let's find out about it again Hermes skills media video prep double click it created this skill for me it says no em dashes spoken English define jargon in line catchphrase you can build anything you can learn anything good we can test the loop a little bit more we can say use delegate task to spawn two sub agents one is researching the LN eval harness another one researching vlm architecture the agent was calling the tool called delegate task and successfully spawned two background agents so you can see it's processing right now over here they're working in parallel cool eventually returned the results to me so you can see that it spawned up some sub-agents to do tasks for me in parallel we've already tried the cron job and the next one is spawn sub-agent that uses clause cli it's my favorite So this is my favorite. Spawn, a subagent that uses the Claude CLI in headless mode, basically using Claude to build a Python script, fetching the top five hacker news stories to Markdown and verify runs. Yep, it firstly delegated the task to Claude CLI. It's using the skill, you can see. Remember, we read the skill here in Autonomous AI Agent Claude. You should be reading this skill. Is this helpful? Let me know if this is helpful. I mean, I was trying to show you more examples in the walkthrough so that it feels more concrete because a lot of people were asking me for implementations from my previous videos. And I feel like Hermes is a really good example. Okay, it created this code for me. And it used the Claude. I created the fetch HackerNews.py. I'll just say, show me the result after running the script. Yeah, so these are the top five HackerNews stories on HackerNews. Let's come check. It's correct. Remember what happened was that we sent the question from the gateway. It went through the agent run, and the LLM basically called up the tool called DelegateTask. DelegateTask delegated the task to sub-agent, and the sub-agent was running in Clawcode, and Clawcode wrote that script. and send it back and then this agent run the script again and then we get the results. This was the hardness in the loop engine in this entire thing. I don't think it self-triggered self-learning skills yet. It only triggered learning the memory. I'm just asking explicitly, did you save any skills by yourself or did you only save memory so far? Oh yeah, but we created that. But we created that skill. You did not self-realize. You need to save skill. You should proactively offer to save a skill after a difficult iterative task with five or more tool calls. So far, we haven't done anything difficult. So we can test the gateway. I can summarize what we worked on today. Hmm, this Chrome job failed. It didn't tell me 10 jokes. It only told me two. What were the rest of the jokes? Don't lie to me because maybe you just created it. Anyways, Hermes team, if you're watching this, Your Chrome job is not updating it unless I ask for the results. So maybe this is something you guys can fix. All right, let's move on and summarize what we worked on today. Because in this current chat, it doesn't know the rest of the stuff here. So let's see if it's calling the actual, remember we say stateDB, memory.md, and skills.md. Good, it actually knows, okay? It says jokes on demand, system check on the OS version, which is what we did. YouTube stats, yes, we did that. Hacker News. Okay, good. WhatsApp here is just a gateway as an interface. It's doing the same thing as the Hermes agent desktop. This is very cool. Maybe this is similar to OpenClaw, but still every time when I feel I can just have a personal assistant handy on my WhatsApp, this is really cool. And imagine if I host this on a virtual machine and it runs forever, I can just basically ask WhatsApp, what's going on? And it's going to reply to me. Oh, I completely forgot. you should set up WhatsApp by typing in Hermes WhatsApp and then just follow the instructions to set it up. It should be straightforward. Last but not least, where is eval? Where is our LLM ops? Remember in our original chart, we had this blue box on the right-hand side called LLM ops. So unfortunately on Hermes, based on my current research, I don't think it has an LLM ops or an eval system. you probably need to build it yourself but it does trace the run there's a thing called trajectory export and logs and it's basically just logging the whole thing but there's no eval if you watched our previous video the eval is basically you can use tools like landsmith landfuse and all these things that can track the entire agent run and some events that happen like tool calls how many times did you call llms did you use some cheaper models to summarize things into your semantic memory unfortunately hermit doesn't do that i'm curious why maybe because it's a local system so you can just customize it yourself it's under Hermes you can click into logs you can see all the logs here you can see the errors you can see the agent runs gateway logs to gateway is the entry point which is whatsapp or this desktop i hope this was helpful i think there's quite a huge amount of work that we covered today, which are exactly what Clawcode can already do, or OpenClaw can already do. I feel that what makes Hermes special, at least compared to Clawcode, is that it's self-updating all these things locally. And maybe this is good for privacy reasons, because I don't see why we should not save it on the cloud. Other than that, everything else should be very similar, right? It runs the loop every time when the agent needs to do some tasks, kind of call all these tools, and you can even schedule a cron drop. And then the memories and the skills are getting auto. updated into the procedure memory semantic memory and the state data will be saved into the episodic memory so i would say this is a pretty standard harness for an agent implementation nothing too fancy but i'm sure they did a ton of work but i feel like this is a good example for how builder these days should be building products because this is becoming a standard right every ai agent tool should be self improving and self evolving. And depending on the user request, you can either keep it local, you can keep it on the cloud, that's up to you. I feel like the skill and memory accumulation is probably the most valuable thing in this, because the user kind of just over time get locked in. And it remembers who I am, which is why like recently, I don't switch to other AI anymore, I just use call code as a semantic memory about who I am. my company information all to manage and what my style of building YouTube videos. But yeah, I think it's a cool framework. You guys should try it out. It's really fun. Cool. I hope this video was helpful. If you have any questions, please leave down a comment. And if you enjoyed it, please give me a like and subscribe. Let me know what else you would like to watch. I will see you next time. Thanks.

I break down Hermes Agent by Nous Research, the open source self-improving agent that crossed 200K GitHub stars in under a year. This is the hands-on sequel to my agent harness, loop engineering and memory system videos: SOUL.MD, the three-layer memory system, live tests over WhatsApp and desktop, sub agents calling Claude Code CLI, and the gaps I found along the way.

Previous episodes:
AI Agent Harness & Loop Engineering: https://www.youtube.com/watch?v=GrNbuWWJYiI&list=PLE9hy4A7ZTmpGq7GHf5tgGFWh2277AeDR&index=2
AI Agent Memory System: https://www.youtube.com/watch?v=mY3bR9qjZr4&list=PLE9hy4A7ZTmpGq7GHf5tgGFWh2277AeDR&index=5
AI Agent System Design: https://www.youtube.com/watch?v=CyLYY_xb5bQ&list=PLE9hy4A7ZTmpGq7GHf5tgGFWh2277AeDR
AI Agent RAG System Design: https://www.youtube.com/watch?v=ZREt9MAozho&list=PLE9hy4A7ZTmpGq7GHf5tgGFWh2277AeDR&index=6

Chapters:
0:00 What is Hermes Agent
0:37 Quick demo: desktop app + WhatsApp
1:17 The full harness diagram
3:00 SOUL.MD: the agent's system prompt
3:53 What a harness actually is
4:31 Loop engineering and Hermes' tools
5:23 Test: cron jobs
6:31 Tests: terminal + browser loops
9:49 The memory system: skills.md + memory.md
11:55 Episodic memory: state.db
12:58 Test: memory save + cross-session recall
13:51 Test: creating a skill
14:31 Test: sub agents + Claude Code CLI
16:26 Gaps I found: self-saving skills, a cron bug
18:16 Missing piece: no eval / LLM Ops layer
19:15 Final take: memory is the moat

You Can Build Anything. You Can Learn Anything. 💪

Launch Full-Stack App Template: https://github.com/ShenSeanChen/launch-mvp-stripe-nextjs-supabase
My AI Agent Startup AutoManus.io is an AI sales lead manager for made-to-order product sales. We embed directly where conversations happen (WhatsApp, email, web chat) to capture inbound, automate follow‑ups, and kill CRM busywork; live with 4 pilots and 2 contracts in motion, pre‑seed backed by Character VC. Focused on US/UK/Germany, raising to expand engineering and scale deployments.

☕️ Buy me a coffee: https://buy.stripe.com/5kA176bA895ggog4gh
- X: https://x.com/ShenSeanChen 
- Sean's Discord: https://discord.gg/tvECErKcFr
- AutoManus Discord: https://discord.gg/5HhcNjCR
- LinkedIn: https://linkedin.com/in/shen-sean-chen 
- Github Link: https://github.com/ShenSeanChen
- Instagram: https://www.instagram.com/sean_ai_stories

Sub Count when launched: 18,303

Top Github Repos
- LaunchMVP NextJS,Supabase,Stripe (1035 stars): https://github.com/ShenSeanChen/launch-mvp-stripe-nextjs-supabase
- ReactNextJSTips (117 stars): https://github.com/ShenSeanChen/yt-react-nextjs-tips
- DeepResearchAgentBackend (72 stars): https://github.com/ShenSeanChen/yt-DeepResearch-Backend
- SupabaseTipsKanban (65 stars): https://github.com/ShenSeanChen/yt-supabase-tips-kanban
- RAGVectorDB (59 stars): https://github.com/ShenSeanChen/yt-rag
- KimiK2Claude (37 stars): https://github.com/ShenSeanChen/yt-kimi-k2-claude
- DeepResearchAgentFrontend (34 stars): https://github.com/ShenSeanChen/yt-DeepResearch-Frontend
- AgenticRAG (15 stars): https://github.com/ShenSeanChen/yt-agentic-rag

You Can Build Anything Playlist (Full-Stack Dev Without CS Degrees)
- Core7: 95% React/NextJS Use Cases https://www.youtube.com/watch?v=vZzFlAjz4rA&list=PLE9hy4A7ZTmpGq7GHf5tgGFWh2277AeDR&index=16
- Core6: 95% Supabase Use Cases https://www.youtube.com/watch?v=Pnoq2tIif3s&list=PLE9hy4A7ZTmpGq7GHf5tgGFWh2277AeDR&index=15
- Core5: StripeNextJSSupabase https://www.youtube.com/watch?v=ad1BxZufer8&list=PLE9hy4A7ZTmpGq7GHf5tgGFWh2277AeDR&index=8
- Core4: AuthSupabaseGoogle https://www.youtube.com/watch?v=AZAU3ClAWWE&list=PLE9hy4A7ZTmpGq7GHf5tgGFWh2277AeDR&index=6
- Core3: FastapiGCPPro https://www.youtube.com/watch?v=u3sfqpzzo-U&list=PLE9hy4A7ZTmpGq7GHf5tgGFWh2277AeDR&index=5
- Core2: SupabaseHero https://www.youtube.com/watch?v=V0kmgkyPsVQ&list=PLE9hy4A7ZTmpGq7GHf5tgGFWh2277AeDR&index=4
- Core1: LaunchAnApp https://www.youtube.com/watch?v=iQ2izvGG4tQ&list=PLE9hy4A7ZTmpGq7GHf5tgGFWh2277AeDR&index=3
- Launch WhereToMeet w/ GoogleMaps, Calendar, Supabase, Cursor: https://www.youtube.com/watch?v=5xMjX1QxGUY&list=PLE9hy4A7ZTmpGq7GHf5tgGFWh2277AeDR&index=2
- Launch EncourageMe w/ Cursor, Supabase, Vercel https://www.youtube.com/watch?v=hnvL4xa2DHA&list=PLE9hy4A7ZTmpGq7GHf5tgGFWh2277AeDR