You Can Learn AI Agent Harness In Real Code In 20 Min | Loop Engineering, Memory, Eval, Open Source

Hey, everyone. This is Sean. So today I'm going to walk you through a real coding example of agent hardness system with loop engineering, eval, and memories. I have open sourced a local first personal assistant that will cover the entire four pillars that matters to an agent system today, which is hardness, loop, memory, eval, and LLM ops, as we mentioned. Many people were asking me if I could show some real examples, so here we are. GitHub is called waku-agent under my GitHub called shanchanchen. I'm going to walk you through exactly how to use this thing. A working example looks something like this. If you have tried OpenClaw or Hermes Agent, it works very similarly. A very quick example would be find me the rest of the World Cup games and add all of them to my calendar. Let's see. So you can see that it checked if you needed some retrieval and then it realized that no retrieval was needed. So now it's running a loop engine here using all the tools such as search the web and it's still thinking looping until it's finishing its task. So you can see that the loop engineering is working around the LLM agent. And meanwhile, you can click into the tools to see what kind of tools are available. And you can click into the loop to see the live streaming of looping. But we can see that in the chat too. So we can come back to this overview to watch the system architecture. In the meantime, it's showing us how much money it's spending. Currently, we're by default using Anthropic APIs. But you can definitely use some cheaper models, like open source ones. cool i think he finished it and then it's asking me if he can access my calendar i'll just allow it so you can see this is a completely local agent that ran through the loop to search across the internet and eventually create the event on my calendar so we can go to my calendar and see okay it looks like we added the norway game for tonight I'm a big fan of Holland. Let's see if you can win. But I live in London, so I also support England. So it's going to be a fun game. I think by the time when we publish this video, we will already know the results. So we'll see about that. So again, if we recap, we have a history of the entire loop that the agent ran through. Apparently, the agent called the Create Event Tool and the Search Web Tool. And you can also check the results here, where he added the World Cup event between Norway and England for today over here. This is a previous event I added with my friend Sergey. So if I don't know who Sergey is, I can just ask WakuWaku and be like, who is Sergey? So you can see that it did the retrieval. It went through the memory system. And then it told me that it went through the gate for retrieval checking. And then you find out that Sergey is my close friend. We can double check that. So this kind of information usually will be stored in semantic memory because it's durable facts. So if I click into that, I have three main semantic memory here. One is that Sergey is a close friend who loves swimming and often cooks delicious food. Raj is a close friend who plays really great tennis and always teaches me great British slangs. And for myself, my channel is usually called Sean's AI Stories and my ex-account is shanshanchan and all of my Chinese account is called xiaobenzunshan. Okay, I can ask you things like, what are my social media accounts? So again, it retrieved the memory again and It ran through the ALM agent. It turns out that there's no tool that's needed, so it just returned the results for me. What you have seen already is a demo of this open source project that runs through the agent harness with loop, with memory retrieval, and last but not least with LLM ops. So if you click into trace, you can see the exact tracing of how much tokens, how much money you have spent, how many LLM calls. And if you scroll down a little bit, you can see the exact thing that activated the retrieval gate in the system that fetched information from the database. and also what are some of the tasks that took the longest time. So the one that I asked about, the World Cup games, took me almost 100 seconds. All of these information are actually stored in the local files called traces. I'm going to show you exactly where that is in the code and how you can implement this step by step. Another way to test this is using a gateway such as Telegram. So what you need to do is that if you want to build a bot yourself, you can come to this agent called Botfather. And literally what you do is that you do slash start slash new bot and then give it a name. I tried many names, but many of them were taken. So my bot is called WakuWaku AI Bot. So if I come here and click on start, I can just ask a question. Be like, what World Cup game is on my calendar? Cool, it says that we have the Norway game. Just for sure, I can ask, who is Raj? In the meantime, we can see that there should be a history. There we go. Yeah, it's real time over here. Okay, it's been synced here. By the way, everything you have seen right now is local. It's stored in your computer, so it's safe and secure with your own machine. I can even add a memory to it. I can be like, please remember that Vincent and I went to Paris to do an entrepreneurship interview together, and now we're both building AI startups and doing great. And if we come back here, you can see that it's using the tools, and I'm pretty sure it's just users. You see? It says that the user was storing the memory, and it called the tool called Save Note. So technically now you should have saved the memory. This kind of thing should be semantic memory. So if we click into semantic, you can see that Vincent has a new fact called Vincent and the user went to Paris together. Okay. Anyways, what I did was just, I just went in and show you how the entire harness and memory and eval and tool calling and loop system works, right? For those of you who are not familiar with these kinds of things, just remember that it's all buzzwords. What I literally demo just now is exactly what these things mean. Okay. And now in the, in the second part of this video, I will walk through this code base that I built. And ideally, if you're interested, you can click on the star and contribute to this public repo together. Are we ready to read the code? Let's get started. Again, we're here on this GitHub repo shenshanchan slash waku dash agent. You can scroll down here to quick start. And literally, we just copy this. Okay. Come to a terminal and paste that in. So I just clone the waku agent. It's trying localhost 778 because 777 was already occupied. but that's fine. So let's come here. So you can see this is a new repo that we just built, so everything is fresh. All right, because we just started a new project, so it's empty. So in that case, we should go check out the project we just run, because it should already have something there. Let's come back here. And if we go to database and click on SQL, console, and hit on run, you can see that we've got all these data tables right here. I'm also just going to switch back to my previous folder. And if I come to .waku and check on database, you can see that it's much longer already. All right. And this is exactly where we're going to store the data. All right. We have the second thing called soul.md. This is basically the system prompt in this open source project. You can just modify this in the dashboard too. If you come to memory and there's a soul file here. And by default says if you feel waku waku, you can often say waku waku as part of the catchy phrases. you can feel free to modify and add something like also say, muchas gracias, when you feel thankful. That means thank you very much in Spanish. You can click on save. If we come back to the code, you will be able to see that it's updated real time. This is your local system prompt, so.md, that will modify your local agent to talk the way you want. We can just test this one more time. click new chat i'll just say you're the best ai agent in the world and it says muchas gracias there we go so what else is in the code we have a memory.imd okay this is where we stored some facts or durable facts okay these durable facts are the ones that are default and you can see that previously we added vincent who i went to paris together for entrepreneurship interview it's also saved here if we come back to the chart that's still the semantic memory if you click into it right You can also edit it right here. All right. Now all the stuff that we run are also in the trace here. All right. If we click into that, you can see there is a whole list of events that happen. Let's take a few look. Okay. It recorded the tools. The tools used, for example, search the web. And then later it shows you the results with the usage of tokens in and out and when it happened. All right. And these are very important things because if you use some tracing tools like LangeFuse or LangeGraph, that basically helps you track these kind of things. But with a local machine, what you want is to save this in your local computer because that's your personal assistant. And of course, if you want to trace this on a cloud, you can upgrade this and connect it to Superbase or just use LangeFuse directly. Again, this project is an implementation of the Harness system on your own local machine. What else is here? That's pretty much it for the memory stuff. We can also check out the eval folder. There are two main folders here. One is called deterministic. Another one is called judge. A deterministic place is basically we wrote out some rules of exactly what we want to test. If we come back to the system, it's basically here. Right after the agent reply to you, you will go through a tracing and then run the eval. Now we're looking at deterministic test. when we have a few tools here is checking if apple calendar was working perfectly and this one is checking if the working memory is working properly because some of the sometimes the eval can be very simple rules and you can just test if it's working right but sometimes it's not true because you might want llm or ai as a judge this was also mentioned in our original system design if you could zoom in a little bit after we trace every single run we will use eval to to check out some very qualitative questions right was this something good or not was it healthy or not so in the code let's see we have anthropic as a judge it basically runs anthropic models to assess some results but i didn't write anything complicated here yet for example this one is for testing the response quality you're crazy guys give a star clone this repo and add your own test and you can check it out all of those will be traced here in the avow okay you're clicking tonight it's everything is in the ops tab what else do we have so we also have skills here Okay, so remember we scheduled a meeting for the World Cup. So we have a skill here that says resolve the relative dates, check the memory context for attendees preferences, call the tool, create event, which will allow the agent to run the tools. Okay, so this is technically a memory too, but it's called procedural memory. It's basically how you expect the agent to act. So if we come back and check out the overview again. when the user asks a question through the gateway it could be a telegram it could be from this chat itself and then it runs through the memory system it checks if it's going to retrieve memory from these kind of things and then it's going to tell the agent what to do all we're doing right now is preparing the right context for the agent okay and the skills is is as i mentioned the procedural memory okay if i click in precision memory you can see we have two skills here one is schedule meeting okay which is literally what we just saw in the code We can modify and be like, always add a heart emoji on the calendar event and save that skill. So the next time if I say, for example, help me schedule a meeting with Sam Altman at 10 p.m. today in London and send it, let's see what's happening. You see it created the event and then it put a heart here. If we check the tools, we can see that it scheduled this new meeting with Sam Altman. If I check my calendar, you can see right here, meeting with Sam Altman with a heart. So basically the procedure memory worked. which was running the updated skill. Guys, this is really fun. You should just keep adding skills to this, okay? You can either, well, I probably need a feature here to let you add it, but you can just add it in the code. Remember, there's a folder called skills, and currently we have two skills here. You can literally just create a new folder here. Let's call it Sean AI Story YouTube Title, and then within this folder, let's create a new file called skill.md. We can just copy this and come here. or you can use clock code to write it for you but we can just do it ourselves let's call it sean ai stories youtube title and the description let's say uh read sean's ai stories on youtube and find out the most popular way to write titles given the new video content so we're just going to keep it simple right we're going to say read the memory from sean's youtube channel and then search the web using the tool search web to confirm the latest five to ten videos and then read the user's brief on the latest video content and return a good title like my own style all right let's see if it works okay firstly let's come to memory and check out the skills which is the procedural memory you can see that there's a new skill here if i just create a new chat So my problem is that it helped me create a new video title for my YouTube channel to promote this Waku agent, which is this current code base. Let's see. So let's see the loop. Nice. It searched the web first because we asked it to. Still thinking. And I got the results. Cool. It read my channel and then it gave me some titles. I built my own AI agent that ruins my life. Meet Waku. All right. This skill is not that good. But you see what I mean. You can just make it better. by continuing to iterate it. And over time, everything will be stored locally. Remember that this entire thing is a local host. There's nothing on the cloud. You literally own this agent, just like Hermes, just like OpenClaw. But it's very simple and straightforward. Literally just clone this GitHub repo and then run it and run the dashboard. And then you get it. You have an agent on your laptop. Oh, by the way, I almost forgot. You need to come to settings and then paste in the API keys for the relevant models you have, okay? So currently I have put in Anthropic key and Gemini key, but you can put in other keys too. Web search, you also need to put in this thing called Tavoli, Tavoli API key, okay? Just to keep that in mind, which is something you need to do as a setup. We have walked through all the memory stuff and the skills, which is the procedural memory. We also walked through the evals. What's left is that in this Waku folder, we have all the code here, okay? The gateway literally has the telegram right here and we also have a voice mode which means that you can activate it through voice. We can try real quick. We can go for uv run waku voice and then now it's listening to me okay so it's trying to capture these these activation word if I say waku waku waku yes so I'm gonna come here hey uh how are you doing today waku waku I'm ready whenever you are what can I help you with I want you to show me what we have set up on our calendar. Get the retrieval. Good. Look at this. Here's what's on your local calendar from today onward. Waku Waku. Event when with. Swim with Sergi. Satjol 11. 5.006.00pm. Sergi. Come on, it's Sergi. World Cup QF. Norway versus England. Satjol 11. 10.00pm. 12.00am. Okay, that's it, Waku Waku. Thank you very much. So apparently the voice was not that good yet. If you guys would love to contribute to this open source, you can work on the voice side. I would really appreciate that. And let's build this out together. But you can see that we can use voice to control this and how exciting is that? And some of you probably realized that in the previous part of this video, you sometimes see Jarvis in this because I previously was calling this a Jarvis project. But then I just realized that, you know, it's not very exciting. Everybody's building a Jarvis. so i just thought about what means what excitement means right and in japanese is waku waku so that's why i wanted waku to be the wake up word for this voice agent i'll continue to build on this as a fun project but the point is that you can control this entire harness with memory using voice as well okay so that was the gateway and the loop codes are here you can click into agent you can see that there's this entire loop agent right here There is an iteration. OK, so by default, we set the maximum iterations to be 10. So it's going to run through this thing across at least 10 times until it reaches the goal for the LM agent. And the model is basically setting up all of these models. And memory code right here is basically how we store the episodic memory, install or load the procedural memory, which are the skills and the semantic memory. OK, the semantic memory. So this is the system. By the way, for the episodic memory, we haven't covered it so far. Let's come back to the chart. Remember, procedures, skill, semantic is for some durable facts, which are constantly being updated by this consolidation task. And then the episodic one is basically a dated event. For example, I was asking it to help me label all the World Cup games. These are all episodic memory because they're dated. On the other hand, the semantic memory is basically... consolidated every single time when it feels like there's some durable fact that it should be saved. And all of these are in these code. You can feel free to dive into it and then make it better if you're contributing to this open source. Ops is how we're going to run the dashboard, release some new prompts, and how do we do the tracing. Runtime is basically it controls all the sessions for every agent run. Tools are very important. We have the Apple calendar tool. we can also add in some mcps which will be some iterations in the future you can even write some notes do some search using the tavoli api okay these things we have already showed you all the coders right here and the main app is here class waku which is literally just a q a agent because it's thinking about how to respond to you after it has digested the model the client system prompt messages tools maximum iterations maximum tokens observers streaming data all these things all right i think this is a very brief walkthrough of this coding example with an open source project to go through agent harness as a whole concept just to recap again it happens when you have a gateway when the user is sending a request and all the agents trying to do is how do we prepare the right memory which could require some tasks so there's a retrieval gate here deciding if we should retrieve or not you know sometimes it skips sometimes it retrieves when it retrieves should it retrieve from procedural memory which is how the agent act like how do you schedule a meeting how do you write a title for sean's youtube channel semantic memory which are some durable facts right who is sean's friends who's sergey who's rash who's vincent right and episodic memory when did sean like exactly build up those dates for the world cup event for talking to sam altman all these kind of things then the agent will decide okay with all these memories and context what kind of tools do i want to ask and run this entire loop until we're like okay we're done let's tell the result back and at the same time trace all the things that happened with an eval system to test things out you know release some new prompt when there requires a new update and at the same time after the reply the agent will consolidate some of the facts into semantic memory. You see, this is an entire walkthrough with real code and implementation on the system design that I have shown in the previous three videos. I really hope this was very helpful because a lot of people ask me, how do we actually implement this in code? And if you feel like this video is a bit too complicated for you, I highly, highly recommend that you watch some of those previous videos on agent harness memory system. and even Hermes agent that'll be very helpful for you to understand this video but for those of you who are technical i hope this was a fun project for you to play around with please give me a star come to this github repo waku-agent under shenzhen chan write out some pull requests let's contribute to this and make this the best ai personal agent on your local machine and i think this will be fun if you have any questions feel free to ask me and leave a comment and subscribe and like the video if you liked it give me a star on github i'll see you in the next video thank you very much

Star the repo: https://github.com/ShenSeanChen/waku-agent I open sourced Waku Agent, a local-first personal AI assistant, and in this video I walk through the real code behind everything from my last three videos: the harness, the agent loop, all three memory pillars, tracing, and eval. It scheduled the World Cup quarter final on my calendar, remembered who my friends are, answered me over Telegram, and even responded to a voice wake word. Nothing runs in the cloud. The whole agent lives on your laptop, and you own it. Step by step, and if the code feels heavy, the theory videos below will catch you up.

Previous episodes:
AI Agent Harness & Loop Engineering: [https://www.youtube.com/watch?v=GrNbuWWJYiI&list=PLE9hy4A7ZTmpGq7GHf5tgGFWh2277AeDR&index=2]
AI Agent Memory System: [https://www.youtube.com/watch?v=mY3bR9qjZr4&list=PLE9hy4A7ZTmpGq7GHf5tgGFWh2277AeDR&index=4]
AI Agent System Design: [https://www.youtube.com/watch?v=CyLYY_xb5bQ&list=PLE9hy4A7ZTmpGq7GHf5tgGFWh2277AeDR&index=2]
Hermes Agent Harness: [https://www.youtube.com/watch?v=LqG1q5NpOBE&list=PLE9hy4A7ZTmpGq7GHf5tgGFWh2277AeDR&index=6]

For those who are not familiar with these terms: it's all buzzwords. What I demo in this video is exactly what these words mean.

Chapters:
0:00 What is Waku Agent (local-first, open source)
0:35 Demo: World Cup games straight to my calendar
2:11 Test: memory retrieval, "who is Sergey"
3:13 Trace: tokens, cost, latency
3:50 Telegram gateway with BotFather
4:35 Test: saving a new memory from chat
5:32 Reading the code: clone + quick start
6:44 SOUL.MD: the agent's system prompt, edited live
7:31 MEMORY.MD: durable facts
8:49 Eval: deterministic tests + LLM as a judge
10:12 Skills = procedural memory
12:55 Test: writing a brand new skill
13:56 Setup: API keys (Anthropic, Gemini, Tavily)
14:31 Voice mode: the "Waku Waku" wake word
16:39 The loop code: max 10 iterations
17:07 Episodic memory and the consolidation gate
18:42 Final take: the whole harness in one diagram

You Can Build Anything. You Can Learn Anything. 💪

Launch Full-Stack App Template: https://github.com/ShenSeanChen/launch-mvp-stripe-nextjs-supabase
My AI Agent Startup AutoManus.io is an AI sales lead manager for made-to-order product sales. We embed directly where conversations happen (WhatsApp, email, web chat) to capture inbound, automate follow‑ups, and kill CRM busywork; live with 4 pilots and 2 contracts in motion, pre‑seed backed by Character VC. Focused on US/UK/Germany, raising to expand engineering and scale deployments.

☕️ Buy me a coffee: https://buy.stripe.com/5kA176bA895ggog4gh
- X: https://x.com/ShenSeanChen 
- Sean's Discord: https://discord.gg/tvECErKcFr
- AutoManus Discord: https://discord.gg/tGsJ2nJH69
- LinkedIn: https://linkedin.com/in/shen-sean-chen 
- Github Link: https://github.com/ShenSeanChen
- Instagram: https://www.instagram.com/sean_ai_stories
- TikTok: TikTok: https://www.tiktok.com/@sean_ai_stories

Sub Count when launched: 21,295

Top Github Repos
- LaunchMVP NextJS,Supabase,Stripe (1047 stars): https://github.com/ShenSeanChen/launch-mvp-stripe-nextjs-supabase
- ReactNextJSTips (118 stars): https://github.com/ShenSeanChen/yt-react-nextjs-tips
- DeepResearchAgentBackend (76 stars): https://github.com/ShenSeanChen/yt-DeepResearch-Backend
- SupabaseTipsKanban (65 stars): https://github.com/ShenSeanChen/yt-supabase-tips-kanban
- RAGVectorDB (64 stars): https://github.com/ShenSeanChen/yt-rag
- KimiK2Claude (37 stars): https://github.com/ShenSeanChen/yt-kimi-k2-claude
- DeepResearchAgentFrontend (35 stars): https://github.com/ShenSeanChen/yt-DeepResearch-Frontend
- AgenticRAG (27 stars): https://github.com/ShenSeanChen/yt-agentic-rag

You Can Build Anything Playlist (Full-Stack Dev Without CS Degrees)
- Core7: 95% React/NextJS Use Cases https://www.youtube.com/watch?v=vZzFlAjz4rA&list=PLE9hy4A7ZTmpGq7GHf5tgGFWh2277AeDR&index=16
- Core6: 95% Supabase Use Cases https://www.youtube.com/watch?v=Pnoq2tIif3s&list=PLE9hy4A7ZTmpGq7GHf5tgGFWh2277AeDR&index=15
- Core5: StripeNextJSSupabase https://www.youtube.com/watch?v=ad1BxZufer8&list=PLE9hy4A7ZTmpGq7GHf5tgGFWh2277AeDR&index=8
- Core4: AuthSupabaseGoogle https://www.youtube.com/watch?v=AZAU3ClAWWE&list=PLE9hy4A7ZTmpGq7GHf5tgGFWh2277AeDR&index=6
- Core3: FastapiGCPPro https://www.youtube.com/watch?v=u3sfqpzzo-U&list=PLE9hy4A7ZTmpGq7GHf5tgGFWh2277AeDR&index=5
- Core2: SupabaseHero https://www.youtube.com/watch?v=V0kmgkyPsVQ&list=PLE9hy4A7ZTmpGq7GHf5tgGFWh2277AeDR&index=4
- Core1: LaunchAnApp https://www.youtube.com/watch?v=iQ2izvGG4tQ&list=PLE9hy4A7ZTmpGq7GHf5tgGFWh2277AeDR&index=3
- Launch WhereToMeet w/ GoogleMaps, Calendar, Supabase, Cursor: https://www.youtube.com/watch?v=5xMjX1QxGUY&list=PLE9hy4A7ZTmpGq7GHf5tgGFWh2277AeDR&index=2
- Launch EncourageMe w/ Cursor, Supabase, Vercel https://www.youtube.com/watch?v=hnvL4xa2DHA&list=PLE9hy4A7ZTmpGq7GHf5tgGFWh2277AeDR