Claude Architect: Multi-Agent Orchestration

Let's take a look at hub and spoke architecture. So hub and spoke architecture is a pattern where one coordinator agent sits at the center and all sub agents talk to the coordinator. I highlighted that word coordinator because you're going to be seeing a lot of coordinator agent. And when you think of like Claude Code, I have a feeling that this is. at least one of the means at least when you're working with sub agents for communication right so this is going to be something really really useful to learn uh it's going to be really fun and something that you can apply um like immediately okay so sub agents never have direct lines to each other so if you have like a research agent over here it cannot directly talk to the review agent review agent it has to go through the coordinator um and so that is pretty clear and the coordinator is going to own the routing so it's going to decide how to route things context sharing so what will be shared so the research agent is not going to be aware of what everyone's doing unless the coordinator passes that information along and it gets injected so um it really won't know uh and any kind of error handling any kind of observability any anything like that okay uh and obviously that would make it really good for observability because now everything's passing through there and we have a choke point where we can check and collect information, right? And so here we have kind of the task lifecycle of like, okay, we have something that needs to be done and how is it going to get executed out? So the role of that coordinator is task decomposition. So break the task into subtasks, right? Then we have test delegation. So who is... going to be working on that problem. Result aggregation. So bring it all back together to produce a final result and decide which sub-agents to invoke based on query complexity. So we have a lot of things going on here, but let's just kind of walk through it. So imagine you are a coordinator, and when given a task, it'll break down a subtask for each of the available tools. we're saying do not do the work yourself so we're basically here defining you know that you are a coordinator and this is what you're going to be doing okay um and then here it's like your coordinator and use your judgment simple factual questions use a single agent multi-step tasks delegate out to a sequential passing the results forward independent subtasks delegate in parallel so we're basically defining okay what does the routing look like so it's not just like static routing it's you know use the routing you need to route based on the use case right which i think is really really interesting and then down below here it's like okay you've gotten all the outputs from multiple agents combine them into a single coherent response resolve any conflicts and make the data pretty so uh that's the most basic thing when we're talking about this coordinator agent um there's a lot of stuff we have to consider when implementing this but we will go and set up a super skeleton one really quick quickly here and then we will iterate on it okay Okay, folks, so what I want to do in this follow along is implement a very simple coordinator agent. So we'll say coordinator agent simple or basic. And in here we will make a new main.py. And I suppose that we could pull up some code. I'm going to see if we can switch over to Haiku and save some credits here because it might be able to do it. We'll see. Haiku. There we go. So now it's switched over to the Haiku 4.5 model. And so I'm going to tell it, create a very basic coordinator agent in coordinator basic main.py. Please follow our general coding. Example would be What was one we did? Decision-making would probably be one. Model-driven.py. And so I'm hoping that by giving it that reference, it will know how to reference that stuff and produce something that's going to be generally okay. But we'll see how Haiku does. I really should run Haiku a lot more. I just kind of stick at Sonnet. And that's my fault there, right? And so we will give it a bit of time here, let it accept here, and then we will decide whether Haiku could even do it or not. and does it have all the components we need? And there it is, that was pretty darn fast. Maybe it needs a little bit more work to do, but we have the start of it. I'll just close out the tab here. Sometimes it helps to close out the tab and reopen it for whatever reason. It's already done. So here it says, I've created a basic coordinator. We have create task, get task status, complete task, task list. And so we have the basic stuff. Let's take a look here and see if it's any good. So we have tool implementation. So tool create task. Task status. And so it's talking about how it has to manage the tasks generically, right? Then down below here, yep, so we have that. Create a task with an optional list of task IDs. Get the current status of the specific task. Mark it as complete. List all tasks as completed. So it seems like it's pretty simple. Your role is to manage and coordinate tasks. Well, a coordinator does that in general, but I guess the thing is like... this is literally it sounds like it's managing tasks create tasks as needed for the workflow check this task status and dependencies complete tasks when appropriate so what i'm trying to figure out here is what is the use case so set up a let's go down here for a second so we have user message so set up a workflow create a task design and then create a task implementation that depends on design and then complete design tasks first this is so generic it's really hard to make sense of it we have our while loop here it brought in the while true so we don't have that max iteration um and then we might not be a major issue but we still might want that in there i probably should have referenced the other code and so just carefully looking here and what we have so we have our tools over here And so I'm not really sure if that really fits our pattern exactly. I'm going to go take a look at our diagram here. What do we have? We have decompose, the routing, and the aggregation, right? So I don't think I see all those steps here. Okay, so what I'm going to do is I'm going to go back to our smarter model here, Sonnet, okay? And I'm going to go and ask the coordinator, or I'm going to ask it to improve our coordinator code. For a coordinator agent, we should have decomposition tasks. Just a moment here. Routing. It says assess complexity, but we have routing. We'll say assess complexity and routing and aggregate results. The use case here is two generic need. a better use case. Okay. And so we'll go ahead and we'll see if it can improve that code. And if not, I might have to write even more detailed prompt. I'm just kind of low on our usage unless the window has rolled over. Let me take a look here. Nope, I still got 15 minutes for my time to roll over over here, but we'll see. And so I just want to kind of see it mimic these patterns here. And so It's not to say that it's not exactly doing it, but it's definitely not that sophisticated, right? Because I would expect there to be a prompt for the routing component here. I'm not seeing it here, right? It does say set up a workflow. So technically, that is that right there. So maybe it is kind of being implemented, but we'll give it a second here. Now I'll rewrite a concrete use case. Technical due diligence, decompose a complex software review. Oh, I don't like that. No, I'm going to stop this for a second here. Stop, stop, stop, stop. I don't like the use case. Oh, it already stopped, basically. So it says breaks the request into five fixed areas. Well, I already got the code, I guess. Let's just take a look here. So here it says a user submits a software system for technical review. The coordinator has a decomposed request, assesses the complexity per area, routes and does that, runs an appropriate handler, aggregates all findings into a single report. So that sounds good. We have tool decompose request, tool assess complexity. I don't really like the use case because I want something that's going to be easy for us to validate and test. And this will be too complicated. I don't like the use case. Can you propose to me 10 possible use cases? I want something that is not super complex, that will be like super computational, but would need... complex routing and choices uh don't implement just suggest ideas okay and so i want to see if we can pick something a bit better if it can't then i might have to uh decide on myself here here's a so job application screener um event planning coordinator bug triage restaurant order customizer i mean i like the travel one that might have to go to the internet i don't necessarily want to do that okay so like with number what would be the subtasks the sub-agents used? Because this is what I'm trying to figure out. And this comes back to, you know, this idea where we have an idea and it's doing a bunch of stuff, right? So like coder, writer, researcher, planner, data, right? So I'm trying to see what we have. And so we go over to here. And so we have request decomposer. So it takes the job posting, resume extracts it out, looks at each criteria and decides the routing. Okay. Execution phase. The actual routing targets. Aggregation phase. So it has two different ones here. So I guess my question is, like, is the ownership? I mean, like, should the coordinator be routing to a decomposer and routing? Because I thought the composer is supposed to own it. in a hub and spoke architecture. And so that's the only thing I'm just making sure, like maybe these are just tools and it's calling out to them and so it still has ownership. So you're right, in a proper hub and spoke architecture, the coordinator's the hub, it owns it. The decomposer isn't a subagent, that's why the coordinator owns its responsibility. So we have that, sure, that looks like something. So the coordinator decomposes, assess complexity, decides where adding aggregates. The spokes just execute. This smells like something belonging to the coordinator versus subagent, does it? need a full picture does it need the full picture to do the its job decomposition is always routing always needs the full picture uh i don't i don't i don't know what you mean by full picture but we are implementing hub and spoke model architecture okay so i don't know what's trying to say there but um maybe it means like just i think that things have to have context you're right okay so this is your right oh yeah of course i'm always right um receives the request calls spokes aggregates the results independent workers each job does one no knowledge of each other that's right okay so for job app application screener now we have keyword scanner deep evaluation red flag detector score aggregator takes all the spoke spoke outputs the coordinator owns calling each spoke with the right input deciding which spoke to call collecting and combining their outputs yes okay and so now we're getting something there Just because it said it did it does not mean it did. And this is me looking at going, oh, that doesn't seem right. Right? And so we will let it go ahead and do that. I'm not getting any more warning. So, oh, this is now using extra credits. So I am over my usage. But I should be okay. You'd be surprised how long or how far $5 will take you here. Okay, so it says it's completed the architecture here. Let's go take a look and check out the code. So now we've got lots of stuff here. Okay, so keyword scanner prompt. You are a resume keyword scanner. Check whether required skills from the job posting appear explicitly in the resume. For each required skill, output one line. Be literal. Do not infer or extrapolate. Report only what is explicitly stated. Okay, and so then we basically have all the ones here, and they're fine. Of course, if we're doing this for real, we would be tweaking this all by hand, of course. And then here we have the actual, I guess they're saying the spokes. We could say sub-agents, if you would, or they have them called as spokes. And each of them are individually calling this stuff. I'm not sure why they're doing it this way. It seems like this could be easily refactored. This seems a little bit messy. Maybe they're doing this so that you could improve it later on. But to me, this seems like this should all be just one function. And then we have the spoke aggregator where there actually is a little bit different. So they do have that there, which is fine. Then we have our dispatch tool. Okay. So basically like where should it go? We have, and yeah, whether it should go there or not tool schema. So what the coordinator hub sees. So we have run keyword scanner. Oh, like these are the actual tools deciding whether they should get triggered or not. That's fine. Then we have our coordinator prompt. So you are a job application screening coordinator. That's fine. Your job is to orchestrate three independent screening agents. Your job is to orchestrate the three independent screening agents and then aggregate the results. You may run three screening agents in order. Do not skip any of them. And so here it's defining. Let's say you have an explicit order. And so obviously there could be more complex routing than this, but this is all there is. Then we'll go down to here. and so here we have our job postings i was going to wonder where this data was because i was gonna be like is there uh are the resumes generated here and they do so or we have our job posting then we have our resume we only and we only have a single resume which is fine alex chen that's interesting okay so we go down here and we're passing that data in it's going through that loop again we have this while true so i'm not sure if that's the best idea to have that while true like there but i will run it and Take the risk. I think it's fine. You know what? I do want a max iteration. So I'm going to go here and just say like the while loop is true. Do you think we should have a max iteration or any other suggestions so it doesn't go on forever? Okay. And so that's... What I want it to answer there we had the max it might just do the max iteration because now I basically told it to do that So here it says a while true loop with no exit condition. I mean there is an exit condition. It's the break right here And then we have a timeout for this use case So a max steps caps is the right fit. That's what it's suggesting Fair enough. Mm-hmm. How's it counting the max steps? So the loop executes only when the condition becomes false max steps and not when the break is hit. So it cleanly catches runaway loops without needing an extra flag. Okay, that's fine. The cap is 10, double the expected five steps, give the model room to retry. I just don't see where it's counting them. Oh, I guess it's right here. Oh, sure. I guess so. But I mean, that's the same thing as a max iteration. So... Max steps is the same as max iteration. I guess it's fine. I mean, I'm sure it will still work. So if it doesn't, we'll find out. And again, you know, you can just watch and see what my outcome is before you do this. If you do not want to waste credits because I've made a poor decision. You know, like I'm loading my thing up with like $5 at a time. So I'm not that worried about small losses like that. So we'll go ahead and go into here. And let's go ahead and execute this. And yeah, I'm not using my subscription. Now we could probably port this over to Agent SDK. And this would be greatly simplified. We might do that later to see what's happening here, but we'll do it right away. So here it says coordinator routes to the spoke. Okay, so it's found stuff. Something's missing. Coordinator routes to the run deep evaluator. Strong alignment with the senior level role with seven years of total experience. Cool. Strong fit. coordinator routes to the red flag detector imagine someone just coded this and this is what's keeping people out of their jobs that that would be a bummer and then we have step two of ten so match keywords strong no flags decision hire Alex demonstrates strong alignment with senior level requirements coordinator for final recommendation for hire so it's recommending it six out of the seven strong no red flags all core required skills present, seven years experience, whatever, whatever. And so we just implemented our own coordinator. Again, the only thing that's really simple, like I'm still not confident about the while loop, but the only thing that is very simple is the routing. But the routing obviously is being handled here. And so, you know, like in that diagram, it just seems like it's a separate step. Like you cut them up and then you do that. And so I'm not sure if that should be separated out, but the point is we did implement coordinator agent. And that's something we could decide later on if we wanted to have an individual step for more intelligent routing. So that's the only thing that I might consider. Like I would probably ask it right now, like if it should be ran twice, but I don't know. I don't want to, because I don't think it's going to just tell me. I think it's going to actually try to do it. And so I don't want to muck with it. And so I'd say that's fine, but just consider that that's an uncertainty. that i have right now and so i'm going to go back a director here just say git at all git commit hyphen m uh basic coordinator i know it's kind of fun i thought the results were pretty good okay and i will see you in the next one okay ciao ciao okay let's take a look at narrow task decomposition so when uh claude decomposes a task it can only delegate what it thinks to ask for Okay, so here we have an example where it says, give me a comprehensive analysis of the EV market, break the user's task into subtasks and delegate them out. And so here we see the subtasks. We have research EV sales figures, research EV battery technology, research major EV manufacturers. So if the initial decomposition is too narrow, entire topics never get researched. It's because each subagent only sees its own isolated context. None of them can flag what's missing. So what got missed? Charging infrastructure, government policies and subsidies, secondhand EV markets, consumer sentiment and adoption barriers, supply chains like lithium and cobalt, grid capacity implications. Okay. So you need to be very specific on the task so it fully covers what you expect. So here it says, give me a comprehensive analysis of the EV market again. And so as the coordinator, when decomposing the task, of course, we're generating out the subtasks, but ask yourself, you know, more information, ask subtasks to cover those gaps, only then begin delegating and for research tasks, specifically consider this information. Now, what's interesting here is like we created our job application thing, but this thing is talking about research. So they might have just a single subagent that just does research. and so the idea is that all these tasks are going to the same sub-agent as maybe separate um instances that are spawned and they're being tasked with doing different things and so this is where you have a little bit more complex routing right or different kinds of routing um and so one thing that we can do to catch weak decomposition is uh because like let's say for whatever reason in here uh this coordinator that you wrote here to help it be very specific, it fails or you just don't do a good job, then you could implement a tool. And so the tool can try to catch it because now when the agent goes and does a task, it's going to say, oh, did you submit a subtask breakdown for review for delegating? Well, then trigger this tool and then make sure right here that you do this stuff. And this gives you a guarantee. you know of this or maybe you want to be a little more flexible what the input is from the user and uh so this thing being decoupled might do that another way that you can fix this problem is at the aggregate level so after you're aggregating the results it can check here and say hey um did you make sure before writing the answer that you uh met these things and so you now have basically two different safeguards for improving over narrow task decomposition. So I'm not sure if this will work in the one that we're building right now or we'll have to build a new little coordinator but we'll go and try it out okay. All right so we are back and what we'll do here is we'll try to figure out this narrow task decomposition. I don't know if it's going to work for our case because for research it's a really good use case but will it be for this one? I don't know. So I'm going to go ahead and just copy all this code here because we already have some that's good. And Claw's going to have an easier time working with tweaking that. Then we're going to have an easier time working off of this. So let's go down to where the main coordinator prompt is. So here it says, your job application screening coordinator, your job is to orchestrate three independent screening agents and the aggregate their results. Run all three screening agents. keywords, deep evaluators, detectors. So let's go ahead and just ask it. Okay, so we'll go here. We'll say, for our narrow task decomposition main for our coordinator prompt, is the decomposition, is our task decomposition too narrow? And what do we need to ask for better decomposition? Okay, because this one's pretty darn simple, right? It's just like, there's these three things, feed it into those three things. Because it's not conducting research, right? It's not going out and looking at large bodies of tax and trying to figure it out. So, you know, maybe if there was like more than one source, then that would be useful. And so maybe that's what we might recommend here in just a moment. I might say, hey, like... you know, assume that you're ingesting more than one source of information, and that might be a better example. But let's see what it comes back with here, and then I'll tell you whether I agree with it or not. Just because it will produce something doesn't mean that it's useful. So we will find out here, okay? Also, I was just thinking about this. What we should have done is just taken the coordinator information and provided it to here with the basic information, because I feel like it's consuming a lot more tokens than it should require. I mean, it's not saying there's that many here, but it is taking... uh some time here and i again i'll just wait but i should have really just extracted out that individual information so let's take a look here and oh yeah we can edit the main file that's fine yep i thought it was done i guess it's not done okay so let's take a look at the problem here spokes are narrowly scoped but appropriately interpretive that's actually reasonable but if you're designing new coordinators well i'm not designing new coordinators but we'll take a look here spokes answers what is x python found but without access to the resume? Spokes answers, what does X mean for the higher? Receives pre-interpreted signals and can make the integrated judgment. So I guess we're trying to determine like, is it fine? So what to ask? So is it narrow? So is skill X listed? Does experience demonstrate the skills X required? So if it's narrow saying like, is it just listed or is it actually telling us so that would be better that's true uh narrow resume only and so this is what i was talking about where we would have more than one type of um information fee but here it's saying it feed in the resume and the job posting for the fit context what granular granularity so one spoke per keyword one spoke per uh decision dimension whatever so this BAL runs both the coordinator of the same candidate. So you can see how the narrow decomposition loses the 50 million requests per day nuance while the better one catches it. Okay so we'll go back up to here I'm just trying to make clear the this thing that we're looking at so narrow anti-pattern. What is x? python found six years three no gaps that's probably how actually recruitment people work the aggregator receives new facts it still has to do all the reasoning but now without access to the resume so spokes answers what does x mean for the higher strong trajectory risks okay so one thing i was thinking of was like you need to cross coordinate this information right so um I would say, you know, one thing I noticed is, you know, can we validate the number of years based on the resume information? Can we mock other data sources that we would feed in where if we didn't do better task decomposition with very specific things to check, we would run into an issue. Because that's, I think, what's going to take it. But that was one example of like, OK, well, if you had to validate how many years someone had experience, you'd look at their resume. But you might also look at projects or references or other stuff. And so let's just see if it can consider other data sources. Maybe we should just want the EV one, because research is a really good one. But I mean, in a sense, we are researching if there are multiple things like maybe they have blog posts and stuff like that but we'll see what comes back here and I might make sense suggestions for data sources okay all right it is back let's see what it's done so we'll go up to here key addition so show activity since 2018 from a get profile let's see all assess skills are above seeing your threshold verified 7.6 years experience okay I mean did it run it again I didn't tell it to run it but I guess what we should do is just take a look at what the new coordinator information is. Your job is to coordinate three independent screening agents and then aggregate the results. I mean, this isn't, this is showing steps, which is fine, but we're not seeing, it doesn't seem to understand what I'm trying to tell it. Okay, so no, I don't think it understands. So what I'll do, just give me a second here. I need to give it an example and I just need to extract out. I'm going to give it a better example here. And we're just going to plot. I have my screenshot. I just don't have the raw data. And so I'm just going to chat GPT or something here off screen. Be like, get me, get me the text. Okay. And just give me just a moment here. Just getting the text here off screen. See, so getting the text and I'm going to feed it as an example of like more information. Okay. So like, go back here so yeah you know you know i don't think you understood uh to improve narrow task decomposition we should be giving it specific considerations okay oh no i didn't say to do that yet okay we'll paste that in as an example right so i don't know if it knows that's an example but i think it might know So hopefully it understands because we're talking about this, this area here. And if this fails, then we could just again, just make it it might even try to change the EV, but we will see what happens here. And wait a moment and see what it comes back with. Okay, so some of your examples general through that. Now, did it change it to EV stuff? Or is it actually changing it to a better part here? So let's take a look here. So what did it change? Let's take a look here. So here's what changed the domain Evie. No, I didn't want you to change the domain. I just wanted you to use that as an example of specific task decomposition. Okay, so there it's already kind of messed up. And I had a feeling that it would do that because it literally did not put eg or stuff in there. And maybe it's just that what we're trying to do does not work for our use case. right maybe but like I still think it is because you are doing research you're collecting information and gathering it but we're just assuming that we already have these things and doing analysis on that information but you know when you're doing broad research and there's a lot of information then it can do do more there so revert the domain name but we'll do self-reflection structure into the hiring coordinator and so we'll take a look at what it has and maybe we still will do the ev example separately I mean, it still has Ds in here, so I'm not sure what it was thinking. Don't say it because I'm not trying to change that right now. Yeah, so like, you are a research coordinator. And I don't know, this is the way that Throbic makes money where you tell it something and it doesn't do the right thing and it's making more stuff here. But we'll wait a little bit, okay? All right, so it's back. And so we say, okay, back to the original domain, right in the pattern. What changed? Narrow coordinator mirrors. The agent basic tells them all exactly the three checks, fix the pipeline, no self-reflection. But that's not what I want. Better coordination, same domain, same spokes, general initial screening angles. What am I missing? Fill the gaps. And, you know, I think it's struggling here. Let's go back down here. So we'll go and take a look here again. Well, here we have the narrow coordinator. So it says here, screen the candidate by running the following checks. And then down below here, we have better coordinator. Generate an initial list of screening angles. Ask yourself what perspective stakeholders or dimensions are missing. Add screening angles to cover those gaps. Only then begin delegating to screening agent tool. For hiring decisions, specifically consider technical skills and soft skills, hard requirements, what candidate has done, and et cetera. After all screening angles are covered, synthesize the report here. So now... I would imagine that it's basically just hitting a single agent. Yes, it is. And so before we had those separated out tasks, right? But just as I thought, it's like in order to do it, the idea is that you say you're a screening agent and then you are contextualizing each one of it. So in a sense, each of these are basically turning that into a specialized one. As before, we literally had three separated ones out. Right? So I think that's what we're getting at. So that is what we want. That's actually good. So we'll go all the way down here. And what we'll do is we'll go, it says both the screen agent is identical both. The only variable is the coordinator prompt. So with a hiring specific checklist of what the coordinator opportunity looks for. So let's go ahead and run that. I believe that's going to give us a better result. Okay. So we'll go here. We'll say Python main.py. We'll run it. And So here it's going through it. So, and we're seeing the numbered values of what it's checking for. Okay. Does the resume demonstrate all the required skills? Does the candidate experience depth? Are there any red flags? Has some Rails experience, though limited? In the 5-8 senior range. No employment gaps or job hopping detected. The career directory is logical, et cetera. And so we have that there. This is the narrow one, right? So fixed checklist, no gap check. Okay, so let's go down to the more complex one. So we'll go down to this one. So now we have way more information. So instead of those three individualized things, and remember, those were three separate things before. Now we have these. I just want to compare the old one quickly here because I just can't fully remember. We go here. Yeah, notice that we have three individualized prompts. and even with the narrow one i guess it's still only passing it through those three and so um that's interesting but anyway so here does the candidate demonstrate mastery of etc okay so we go down here and i guess we can't really see the individualized results so that'd be something you might want to do is like output all of them and then save them and then save the generated uh final one to exactly see what it is so core stack matches excellent okay risk and gaps questions for interviews final final recommendation now we have a maybe alex is qualified candidate but has some gaps for a true senior role hire if your team values this passive whatever bottom line alex is a strong back-end engineer and then we have coverage and so it's way better in terms of its information but really to test this you'd actually have to um you know create sample data right and test it and then and then adjust and say hey look this is not how I would have judged it right based on that information but this is the example that we wanted but really that works when you know there's a generic research agent and then these individualized things are going in and kind of helping to specialize that research agent for its task but yeah that was cool all right let's talk about dynamic selection so the idea here is that when you have your coordinator and you have sub-agents you might find that if you run the entire pipeline for every single possible spoke in a sequence that you are consuming as much as you can and so with your coordinator you probably want to tell it to think about what kind of pathing it needs or what kind of routing it should have and give it ideas of kind of routing that it can perform under certain circumstances so that it's doing exactly what it needs to do even here it's saying like you know only invoke it if it makes sense um and a way we can catch that problem so if we have a a poorly designed um dynamic selection system you can set up a tool just like how we talked about with the narrow task decomposition you could set up a tool that says hey did you do a good job here you can do the same thing with the tool as well and I mean it's really going to depend depend on what you're doing but that's something that you know we'll want to consider okay hey folks we are back and we are going to try to do some dynamic selection so um you know for ours uh our our job application um basically we are taking in um information from from one thing but basically uh it's just checking everything and so you know the question is like can we even think of any kind of select dynamic selection that would be need to be performed for a job application because i feel like it would be more if you were to ask certain questions to the agent along with this coordinator then that's where I would want to choose different types of pathing. And so I'm not exactly sure. We'll let it help us think of an idea. But I do want to remind folks that we are just doing this to learn. If you are doing this for real, write these things by hand yourself. Use your brain. That's how you're going to get the best result. Garbage in means garbage out. So just because this thing works doesn't mean that this is well designed. We're just going through this to learn these concepts, right? Okay. I'm not saying that this is the best. But anyway, we'll go ahead here and make a new folder. This one will be called dynamic selection. Okay, and I'm going to go ahead and make a new main.py file. And I'm going to go ahead and select this code. And we're going to copy this. And we'll paste this into here. And so I need to give it a concrete example of what we're talking about for dynamic selection. I do not have my text here. I put it in there, I'm going to get chat GPT to extract out just how we did with the narrow, narrow one. So just ask it to, you know, extract out the text for me here. I'm just doing this off screen here, because I need to give it a practical example, and try to describe what we're doing here. I'm going to CD back a couple. And we'll open that up. And so we'll go here and just say, you know, I want to implement dynamic dynamic selection for my coordinator so that it's not running the entire pipeline but trying to choose the best things to run based on use case here is an example of good dynamic selection where we have different pipelines. Okay, you can use to help you. Okay. And the other thing is like, edit the dynamic selection main dot py file. Okay, so it's going to go off and do that. And we'll see what it comes back with. Hopefully, something that is useful. But you know, we don't kind of guide it and say like, these are the use cases, you know, I'd be surprised if it doesn't come back with anything good, but we will try here just for learning purposes, okay? All right, let's come back with something. Let's take a look at what we have. So dynamic coordinator. Oh, we still have that narrow coordinator in there. We should really remove that out of there because it probably is confusing. We now have like three coordinators. I don't want to have three. So we'll go here. I'm just going to tell like... Look, I only need a single coordinator prompt. So I'm being lazy here. If we don't need the other ones, we can just delete them out, right? We have this narrow one. So we know this one is not something we want. So we'll take that out. Then audits the gaps before delegating specific domains. And then here we have the dynamic one. So we'll take this one out. Okay. Look at that. We wasted no tokens. There's no reason we can't do that. We don't have to prompt everything. Then we'll go down here and take a look. So this coordinator reads the roles, then decides which dimensions actually matter. So routing the logic. So strong technical match or whatever, whatever. Let's take a look and see what we have. So routing guidance. Adapt to what you observe. Don't apply mechanically. So simple factual match. Skip keyword scan. go to straight to this non-traditional background transfer skills oh this is cool i like this never invoke a screening agent unless it answers a real question so i think that actually worked out perfectly that's a great example of of that and so we'll go ahead here and i'm just going to go and run it so we'll cd into dynamic selection this has actually been quite fun um is it useful i don't know depends on what you're building And yeah, we don't have the narrow coordinator. So I'll go here, just make sure narrow coordinator. I want to get rid of these other ones. I don't want to waste all that here. And so I'm just going to go back a step and just say, oh, this should be fine. Let's just do that. I didn't realize there was more to rip out. I think it'll still work though. This is all three coordinators. There's only one though, right? Because I ripped them out. So here it says, describe the most complex screening angles delegated. Okay, and the only way to really know if this is different, what happened here? We have narrow QS. We still have some of that remaining code there, so it's just some of this stuff. And so I'll go ahead and try that again. I'm not sure if that will work, if it's just a single item, but I'm hoping that it does. But here, my best guess is that it's choosing exactly what it needs, because if we go back up to here, that's what it looks like it's doing. Oh, my goodness. So I'm going to go back a directory here, because this is very frustrating. Remove the better. I only have a single coordinator. I remove some of the other code because I only really need a single coordinator here. Can you fix the code? You know, what's funny is that sometimes like I will type things and every single letter will be wrong and it still knows what I'm saying because like it does the off shift, which I think is really cool. As someone that's dyslexic, as long as it understands me, I love that. Yeah, and so I'm just asking it to clean it up. I just want to make sure it's in a working state before we move on here. All right, so it thinks it's cleaned it up. We'll go into here again and we'll run this again. So screening angles detected. What I'm trying to determine when it runs here is like, how is it selecting stuff? So in your current role at the FinTech, what is the scale of your system that you worked on? You transitioned from this, have you designed or refactored it? So what it looks like it's doing is it's actually generating out possible angles based on this information. And so it's literally creating dynamic routing on the fly. So it's not like here is a list. that we had before here, but literally like here are things that you can check and then choose what you want to put in here. So it's not always applying the same thing. Then we'll go back up to here. It's still conditional maybe, but yeah, we're getting something that is, it's again, very interesting to see this play out. Again, you know, we don't know if it's actually useful, but it's fun to see the system working. And there you go, okay. Let's take a look at partitioning research. So if you give three research agents the same brief, you get three overlapping answers and wasted tokens. So if you're trying to paralyze things, right, and say research CV market, research CV market, and they're all doing the same thing, that's going to be nonsense, right? So carve up the scope so each agent owns a distinct slice. And so here we are seeing partition information. where we are creating structured data and we're providing detailed information like topic, cover, excluded, things like that, and providing that information there. And as per usual, we can create a tool that would check and make sure that we're dividing the research scope into non-overlapping assignments before delegation. What's really interesting here is that it's making a structure. And I mean, you know, in the last thing that we did, technically it is already kind of assembling its own way of doing stuff, but... I suppose what we could do is we could generate out into partitions in this sense and make sure that it's even more detailed in terms of what it's covering as an intermediate step to make sure that it's not doing the exact same thing. But this one's more focused on very specific things that it's researching. But yeah, the question is like, does our current one, even if it's not the exact same one, is it having overlapping tasks? Because that's what we don't know. And so maybe putting a structure with partitions. might help. But we'll have to experiment. Okay. Okay, so let's see if we can implement partitioning in here. So what I'm going to do is make a new folder here called research partitioning, partitioning. And we'll go ahead and make ourselves another new main dot pi file. We're having lots of fun here. And so we'll grab this wasn't this one is the last one we worked on, right? Going to grab this here, copy it. And Go all the way down. And oh, wait, no, no, this one's empty. Here we go. Okay. So now we are back with our dynamic one. So here we have off screen here. I just told it to extract it out. Right. And I'm going to say here, like, you know, I want to make sure I want to make sure my research, what are they called? What did they call them here? my research agents aren't wasting credits tokens by having and time by having overlapping tasks. And so I would like to have another step where we have partitions. And I mean, like the thing is, like you could manually make this stuff, but I'd rather this generated out so makes easier for us. So we have partitions have a step where we generate out partitions based on a JSON structure. And then we can determine if there is if they are truly not doing the same task, make sure to print out the structure so the human can see it on the run of the coordinator agent update the research partitioning main.py and here is an example of partitioning from a different use case okay and so I'm going to copy this over bring it on over here right and I'm hoping that this will work Right. But I mean, like this, it also could just be like a static way. Like if you were just statically building a research agent, that this would be a means to which you could do it. And you don't have to delegate so much out to the agent, if you will. But then now we're kind of relying more on code driven logic, but you can mix them by the way. We didn't, we didn't mention that, but you can take a hybrid approach where some of it is the coordinator and some of it is code driven. There's nothing wrong with it. There's no rules here, folks. There's no 100% that. It's what you want to do, right? And so we will see if it can come up with something here, and then we will review that code. Okay. Okay. Let's take a look and see what it thinks it's doing here. So narrow. We still got this language like narrow versus better. I probably should get that out of there. I really should be taking this out so that it's not getting as complicated. Here, so screening agent prompt. You are a specialist hiring analyst. You will be given specific screening questions about a candidate. Answer the questions two to three focus sentences. Be concrete and specific. So really changed it in this case. Oh, no, no, this is fine. This is still the same. Runs a specialized agent. Calls once per screening, et cetera, et cetera. I don't want... Yeah, I don't want to muck with this one. Oh, now I don't know. Did I break it good? I think I can just go ahead here and discard the changes. I think it'll be fine. Okay, so we'll go over to here. This is the one we actually wanted. And so it still has that logic in here, which is kind of a problem, but we'll see if it actually is an issue. Because we only have one, right? I'm just going to remove it. I just don't want it to get confused. and i don't want to explain any of that here so i'm just going to take that out so let's take a look here it says for both coordinators so spoke system prompts i'm just going to take that out it keeps talking about like okay and now let's take a look here so you are a partitioning uh a screen partitioning planner given a job posting resume output a json array of non-overlapping screen partitions each partition object must have an agent a scope Rules design partitions so that together they cover all relevant hiring questions. No two partitions may share the same cover aspect. Only partitions that are genuinely needed for this candidate. I feel like we lost information here. We have, oh, this is the planner part. Okay, so this is actually a separate part. Okay, so that just generates the partition. All right. And then down below here, we have... The actual dynamic coordinator. So you're here invoke exactly one screen partition call per partition, no more or less formulate the questions for each cell. Do not invent additional screening angles beyond the partitions provided. They were designed to cover all relevant dimensions without overlap. And so, I mean, like that's another way where it's just specifying it in a different way, but I guess it's generating out the partitions. So in the other one, we literally listed out possible things and here it's generating them out. Oh, here it is. Is this good? Because before we had a list, right? So we go back over to our, we'll just split this for a second. And we go back to our dynamic selection here, right? And so here we had this, but we lost our routing guidance. So this is what I'm going to ask. I'm going to resume the last conversation we had. I'm going to make this a bit larger here. No, so we don't have it anymore, unfortunately. Or maybe I ran it in a subfolder. That's my problem. So I'm going to go here and ask it like, we have a dynamic, what is it that we have? We have a research partition so that we don't have overlapping researchers doing the same thing. Did we lose selective routing? based on task and do we need to bring that back in while preserving our partitioning okay and so I'm going to go point to dynamic selection has the original prompt that had routing and then here we have research partitioning is our new prompt with partitioning but the routing was removed and so how would it know to do routing like do like how would it know to choose the appropriate dynamic selection okay and so that's where I think there's a bit of an issue okay because sure like we now it will generate out that structure and things but how does it know to drive what to generate because it doesn't say that right and so then the both selects the angles matters and delegates it has routing that skips beyond the scan and strong matches now owns the selection step but it has no routing rules it just generates non-overlapping partitions will guidance which uh on which one will skip and so here okay i'm sorry tab here okay you are moving it back in But are you thinking about how it will work? Or are you just dumping it back in? Okay, because I could have done that. That doesn't mean that's a smart approach. And maybe that's all we need to do. But let's take a look and see what it comes up with. I'm telling you, these things aren't smart, folks. You got to think for it. So here we go. Fair point. Let me think about it. So we have partitioning that sees the resume. and then the dynamic coordinator which uses the partitions. Where does the routing actually live now? So in dynamic selection the coordinator was doing two of one things deciding what to check and calling agents and the research partitioning now splits routing, partitioning planner's job and then the coordinator's job. So the routing rules do belong to the partitioning planner but they need to be reframed. In the original one it says never invoke the rules unless. Here we should say only create partition if the rules govern partition creation. The coordinator in the new architecture should say as is it's correct to be dumb selection because the decision was already made upstream. Giving it routing logic again we create a conflict of two places fighting over what gets evaluated. The partition planner currently says only include the partition genuinely needed. That's vague adding a specific heuristic routing. I mean like I would have thought and you might agree with me is that in the dynamic coordinator we had that list. uh but the thing is is like it was saying like okay if you do this then then do this or do that but maybe the problem was is that when we looked at the ev research it literally had pipelines where in this dynamic router it just had things that you could choose from that you might want to consider okay but it wouldn't run them all so here it says we receive a set of pre-planning partitions as json invoke exactly one screening agent uh-huh okay well let's just see what we get Okay. I'm not sure if I like it, but we're trying here, right? And we'll go main.py, we'll run it and see what happens. So we have, we have core stack proficiency, assess REST API capabilities. Okay. So we have here core stack proficiency, evaluate mastery of required technologies directly matching the job stack. Access REST API design capabilities. evaluate exposure to scaling patterns and nice to have technologies, confirm senior level experience. And then here we have screening angles, Delicated. Does the candidate demonstrate mastery of required stuff? Okay. And here we're getting partials. So we have a maybe recommendation. Alex is qualified to mid to senior. And we have different coverage. So I still don't know if this is better. I mean, like... we should be dumping all these logs out and then comparing them and then and doing stuff so obviously we're just trying to meet the requirements of uh learning this stuff and kind of having a sense of it but is it good is it is another question that will take more time and i'm going to keep repeating that because i just want you to know just because we're doing it doesn't mean it's great um and you should be thinking about like okay if i had these three four different ways um you know determine usage determine outcomes have your examples don't have them for you here that'd be a lot of work for me to set up for you um but uh no it's interesting trying to try out these techniques and apply them okay let's take a look at a refinement loop so the idea right now is that everything's been one shot the idea is it goes through it it produces an evaluation and then then it's over but what if we could feed it back in the loop and refine it until we are happy with it and that's the idea behind refinement loop to make our research system really really good um so if you look at this prompt here for our coordinator the idea here is that we are telling it that we could have up to maximum four refinement iterations and that we are going to redelegate the information back into here and so you'll notice here we have like an evaluation coverage and when to submit final and creating the synthesis and things like that and so we will go ahead and try to apply refinement loop to our our agent okay Hey folks, it's Andrew. In this video, we're going to implement our own refinement loop. So what we'll do as per usual, I'm going to go ahead and make a new folder. This will be my refinement loop. Okay. And then what we're going to do is we're going to go ahead, let's just grab our code here, main.py. We're going to go grab our last one, which was research partitioning. Because we're building off of it every single time, trying to make this thing a little bit better. and we are going to implement the refinement loop i need to extract the text out because again i i don't have it on on hand but let me go grab it from that slide okay there we go i grabbed it and if you want to grab it too all you got to do is take a screenshot feed it to clod or chat gpt and extract it out folks um because you can you can make sure you do that okay it's not hard just build up those skills okay uh so i'm going to go ahead here and just cd out here we're going to go into our clod and um you know i need to implement a refinement loop in my agent for research partitioning main here is a example from another use case you can use as inspiration as guidance. Okay. And so I'm going to paste that in there. And so the idea though, is that with that information, I'm hoping that it can develop that refinement loop in here. So we will see what it produces. Okay. All right. So in here we have changes. Let's take a look and while it's still trying to edit stuff. So yes. And let's see what we have. Okay. So It is bringing in evaluate coverage. Okay, so we have that. Submit final. So it's setting different states based on whether, you know, hire, maybe, or pass. Only call this when the evaluation confirms sufficient coverage. And final recommendation. So we have that in our loop. Here it is adding the evaluation agent. Okay. And we have some tweaks here. So we have initial screening. Invoke exactly one screening agent call per partition. Formulate each question. That's fine. Phase two, evaluate coverage. After all initial partitions agents have reported, call evaluation coverage with plain text. Then here we have refinement max reiterations. If the evaluation coverage returns sufficient falls, invoke screening agents to fill only identified gaps, call submit final, et cetera, et cetera. Do not call the submit file before evaluation. uh if it's only once okay so here it's obviously done a lot i'm kind of curious to think like maybe it's just like you're brute forcing to make it either that you really want this person you really don't want this person it'd be interesting to have a larger data set like let's say 100 applicants and you ran it through and to see if it just skewed it to one location or one side or not um but it would be very interesting to find out but we'll go back up to here and so we can see the changes And let's go and run this thing. Notice as we are progressing, it's becoming easier and easier for us to update our agent. And so far, we've been just using the Anthropic SDK, not using the agent SDK. The agent SDK is awesome, but we will just continue on here. It'd be interesting to convert it over and see what the code looks like. And we'll probably do that. But let's go ahead and do Python main.py. And we'll go ahead and run that. And the idea is going to run, it says dynamic coordinator. Obviously, it's the refinement one. We don't change those names. And so here reads candidates first routes to the relevant checks only. So evaluate depth, access to database caching, verify API, confirm senior level experience. And is the candidate senior level? Okay, great. So now we're going into iteration one. Okay. So we have coverage score, code quality practices, no evidence, et cetera, et cetera. and so it is going again here asking questions they are i think they are different questions it's hard because we have the this up here right and then down below oh look the the coverage score is going down now interesting and so we are done and over with we'll go and look up here so did two iterations and their score went down so yeah that's iteration loop is that good I don't know it takes a lot of work to evaluate this stuff we would spend hours hours upon hours tweaking this to figure out is this valuable information is our data set good etc etc there's no magic here folks we can code these out very quickly but to make sure they actually work good as a different story I'm going to keep repeating that because it's true but that is what the refinement look refinement loop looks like okay okay folks let's take take a look at observability so the idea of having this centralized coordinator is the fact that everything's going to pass through it okay so no matter who has to talk to who it's going to pass that coordinator and because of that the coordinator is at a choke point where it can observe um anything and catch any kind of errors because it is coordinating stuff but when you do not have that then everyone is just communicating with each other and you can't observe what was said you can't catch errors consistently you can't control what information crosses boundaries but the coordinator can do all those things and so we are using the coordinator pattern um do we have observability that's a good question so i would say let's go back to our thing and see If it is working, I would probably ask, like, hey, can it actually route to other ones? And is it capturing information? I know it already probably is working this way, but let's confirm and go back to our code, okay? Hey, folks, we are back. And I'm going to make a new folder. And this will be coordinator observability. Because the idea here is that our coordinator should act as an observable layer. And so I want to make sure that it actually is doing that. So let's go back here. uh go into um formal type clod here right and we already have the refinement loop over here so i'm going to go ahead and grab all this code okay i'm going to grab all this code and we'll make a new file called main.py you're starting to get the pattern here what we're doing right very repetitive but it really is good in iteration so what i want to figure out is do we actually have those values is the coordinator acting like a coordinator so I'm just thinking about this for a second. So what we want to do, so we're going to say I have an agent, a coordinator agent here. So we'll say coordinator observability here. I'm going to put this into plan mode. Here are the questions I have. Is my coordinator operating from observability level, observability and I'm trying to think of what like controlling flow of information, you know, is it managing, you know, like is it, I'm trying to think of it here. I have a coordinator agent, right? Here are the questions I have. Is my coordinator operating with an observability layer? So we can capture any errors, all messages that are being sent to our spokes sub agents is it controlling context and what is passed uh to my um uh spokes and only those sub agents can talk to the coordinator right is there something i am missing to make my coordinator a good coordinate coordinator right that's what we want to know and we'll go ahead and hit plan there you're thinking well you know you're just writing whatever out but that's that's a fine problem you can't make perfect prompts here folks i mean you can make better prompts yourself and spend time engineering them but this is good enough to start getting some information so let's see what it comes back with and i'd be curious to uh layer something here uh there so i'm just curious what we can do um to make our observability better um but we will wait for that generation okay okay so let's take a look and see what it thinks So we'll go all the way to the top here. So the user's coordinator uses a partition-based hub spoke pattern for job screening. They want to know, does it have proper observability? Does it capture all messages in spoke? Does it control context past the spokes? Is the spoke coordinator communication isolated? And what is missing to be a good coordinator? So it says observability is weak. I absolutely agree. We haven't done anything for it. Could we do something better than just... Oh yes, it's only print statements. I agree with that. Error handling anywhere. It says JSON loads. Yeah, that makes sense. API level tracing. No token counts, no latency, no request IDs. Yeah, token counts would be really useful. No persistent audit trail of all output and ethereal SD out. We have message capture partial. So delegated list. We are printing some stuff, right? Because I told it to do that. Tracks questions sent to spokes. Spoke responses are never stored. They go straight into the tool results and are lost out of the coordinator. No record of partition. Each question addressed X. No log of coordinator's own reasoning. Text between tool calls. Every spoke receives the full job posting. And so saying context control loops. Resume regardless of its partition. I mean, it would need it anyway, right? Like all of them need it. So we're not really giving them separate data. Regardless of its partition scope, so cover excluded is only advisory only passed by Jason to the coordinator prompt but not enforced at the spoke level. The coordinator could ask any question. No mechanism validates the question stays within the assigned partition. Fair enough. Spoke isolation one direction one directional line forces spokes are stateless functions called by the coordinator. No cross No cross spoke communication. So that's good. But spokes have no awareness of their assigned partitions. They can't reject out of scope questions. Oh, that's actually a good point. No, only these spokes can talk to coordinator mechanism. It's just a single plane one. So where are our gaps? can't debug or audit the run, silent crashes on failures, can't replay or inspect what's wrong, coordinator doesn't know if all dimensions are covered mid-run, recommendation may be issued before all angles are addressed, spokes are all data even if relevant, all partitions even if some obviously irrelevant, a single pass cannot fill gaps during it. And also our data, like if really if we may want to make this better, we would be sending like different payloads of different information to specific research agents, but right now everyone's getting the same job posting. So here they have recommended fixes. So structured logging with timestamp levels. That seems good. That's fine. Sure. And it's going to log that out. Error handling. So wrap the JSON load, generate the partition, persist spoke inputs, outputs, extend beyond the tracking, add coverage evaluation tool at explicit gates, force the coordinator to call submit final. And so these are fine. The only challenge here is I don't I feel like there's a lot of tasks here. So I'm just going to go here and say there is a lot of of work here. I am concerned you will not be able to remember all the tasks. Can you create this plan in a readme with a task checklist? And can you check off the tasks? as you complete them. Okay and so the goal there is to help it out a bit. That's not exactly spec-driven development but the only thing is like if we really wanted this to drive and be really good we would want something that would clear context each time but we're not set up that way. I'm not here to roll a small spec-driven development little thing for us here so that's totally fine and read me with tasks and checklist. As long as it knows where it's doing that, that's fine. But where is it going to put that file? I'm just going to trust that it can do it. Let's go ahead and let her rip. And so the idea again here is to improve observability, and we will see how that goes. Another thing is, I would imagine that if you were to put this into production, you want to productionize it, you might want to contain these subagents into containers. And then you might want to use OTEL as another observability layer. that's how i'm kind of thinking about it but we're keeping it all monolith for now and we're not going to overly complicate it at this stage um and i'm going to let it go and burn away all my credits look at that 6.2 thousand credits wow let's go over here it's like the worst time to do this people are like there's uh on twitter they're like oh it's down and the usages are gone and stuff like that it's me it's me i'm the i'm the problem so we go over here and right off the bat like we are uh Oh, it resets in nine minutes, so that's really good, but we're only 33%. Well, let's burn, burn, burn, baby. Though my week is getting up to use very quickly there. But anyway, oh yeah, it's going up. So yeah, we're going to consume tokens like it's nobody's business, but I think it'll be worth it for this stage of the thing as we are continually refining it, okay? That was fast. I feel like it should have taken longer than that. All six fixes are implemented. I mean, I would rather it have been more granular and clearing context between it so that we would have better stuff, but it's fine. So is my coordinator operating with observability? No, it had only print statements, et cetera, et cetera. What was I missing? Error handling, mid-run gap detection, no exit gate. And I'm not even saying this is the best, but... you know it's pretty good for us throwing things here together let's see if we can see what i mean probably just tell us what code changes were made i suppose that's the easiest way to check and um i'm just going to go all the way up here let's take a look here so we've added logging okay and we are implementing the logger here it says scope to each partition person agent name uh is the names the question belongs to from the partition jason Okay, so it's being very particular to make sure that it's scoped. That's good. Evaluate coverage. So mid-run gap detection tool. Evaluate whether the screening fines are efficient to make a confident recommendation. Confident that all partitions agents have reported. Return a coverage score, etc, etc. Okay, submit final. Explicit exit gate. Submit the final hiring recommendation. Call this only after evaluate coverage. Fair enough. So go down below here. Mm-hmm. Fix error handling. So here we have... Here's down the error handling down here below. Fair enough. That's very basic. That's not really that important. And then we have the screening agent. So we are seeing... Oh, just scope it in the boundary, right? So making sure that it's scoped. Fair enough. Here we have rule changes. And so it's about passing that information and it's talking about that evaluation coverage in the final recommendation. Okay. And then we got logs, logs and more logic. Now this thing is pretty wild. I would probably want to take it farther and refactor it, but I don't really want to, like, this is just, this is a mess. Like, this is not how you should have your code base, but I don't want to go overboard at this stage. I just want to make sure that this works. Okay, so what we're going to do, because I'm expecting logging to appear, right? And so I would probably say, like, we could just run it. But the other challenge would be, like, we need actual ways to test that this stuff works. So probably what would have been better, but it would have taken, like, this would have took an hour or two, and you folks don't want to wait around that long to test that. But what I would have done if we had the time and you wanted to go through it, what I would do is I would stage... examples and and I would want to see if like we could pollute the context between agents and make sure that it is only receiving proper questions and it rejects it and those would be things that we test for so we really are skipping a lot of that stuff and that's stuff that you would still have to do but just because I'm not doing it here doesn't mean that I wouldn't do it it's just I'm not doing it because you folks don't like when I make like two three hour videos even though that's the actual effort for the work and i can't really just you know fake that along right so we'll go ahead and we'll run that i think it's about observability wrong which is fine and so what i'm interested in is do we get any logging where's our logging i don't see it okay i mean it's just going to st out so it's not logging to anywhere in particular which is fine uh so you know i'd probably just have it logged like into a log directory and that's the thing that might be missing here okay And I'm just going to wait for it to run. Oh, there we go. There it's done. And so we have our final information. Did it call that final evaluation step? Yeah, final recommendation. There it is. So there you go. That's all it took to improve it. Definitely better than what we had before. But yeah, I would probably want to refactor this. So that's like, you shouldn't have one big dumb file like this. And so we might do that in a separate video, especially if we want to convert it over to the agent SDK to compare. That might be something we might want to do. Okay. But yeah, now we've added observability. Hey, folks, it's Andrew. And in this video, what I want to do is I want to refactor our coordinator that we've been working on up to this point, as I'm going to want to port it over maybe to Agent SDK to just take a look and see what that looks like. And so we're just going to say coordinator refactor here. And I'm going to go ahead and grab the code here. And we are going to ask it to refactor. and let's just see if we can make this a little bit more maintainable okay if you are not a programmer you might not know that this is not good code okay and it like it works for this point that we've been able to hold this all into memory but if we came back later we wouldn't be able to really make sense of it and just because the agent can make sense of it and summarize it to it that's not good enough we need to make it so that it is more human readable um and that is what we are going to do so i'm just going to uh cd out of here and we're going to go into clod and uh we are going to get some refactoring going on so what i'm going to do is i'm going to make a new file called refactor um refactor md and i'm going to go say refactor um tasks so this is this document um is the tasks i want completed to refactor this, our coordinator agent. Currently, all code sits in the main.py, and we need to break it into multiple files. Okay, so let's go ahead and start making some tasks. I'm just going to make my observations of what I don't like. So the first thing is, the prompts. So all prompts should be stored as, all prompts should be stored, all prompts should be stored as markdown files in a prompts directory. Okay, so that's step number one. The other thing is like tools, see how tools is very wieldy. So tools should be individually defined as their own files in the tools directory. We should have .py files for each actual tool code and the tools dot the tools. I mean like can we, it's just JSON right? Can we? I don't think there's anything special about this and the tools JSON. for the long tool right I think it will understand what that is for the gets passed to create so that is definitely something I would like fixed what else what else um during partitions we do have the partition system so say partition generation should be in lib as its own file. That's something else I would do. That's a function. That is that. Run coordinator. The logging is inconsistent. I don't like how the logging is. So we should have a logger that refactors all the logs to be consistent in a file called logger. in our loggerpy in our lib directory. That'd be another thing I would want. Yeah, so I think that's a start. And so I'm going to go ahead and just say, coordinator, I want you to refactor my code based on that markdown files tasks for the... main.py in the coordinator refactor. Okay, and so we will let it go do that. And we'll see if we can really reduce and organize that code base because it should be really easy for us to read, right? Like I know we can make sense of it because we've been carrying forward it, but we really need to be at 100%. Like, yes, absolutely. I know what I'm looking at. Okay. So I'm just going to accept everything that goes along here and then we will take a look and see what we have. So it's already off to the races. We have our prompt, our partition planner, our screening agent MD. For me, like I would probably want these to be templates that you can inject stuff into, but there's no reason for that right now. We don't really need dynamic injection, but it's definitely something that would be interesting to do. We have our tools directory. I think we only have the one agent here and then we have our tools JSON. So that is... uh working out pretty well so far it's going pretty quick too man these things are getting really really better here we have our logger and then we are going to have our partitions up high i really don't like that we have constants i do not like using constants whatsoever i think they're just it's just bad bad code uh but we'll continue on here and i'll check it out in a moment i'm going to just check how my usage is going and uh it doesn't matter just reset i'm back at two percent look at that lucky me Okay, so we are just chilling out here waiting for this to generate. I'm going to pause here and we will come back in a moment. I think it might be done. I didn't even really wait that long. And so we'll go up here and take a look. And so we have our main.py, we have our prompts, we have our tools, we have our libs. Let's go take a look here and see if this is reduced enough. Okay, and I probably should have told it to check box off these, which is fine. So I'll just go here and just check them off myself. I just didn't feel like telling it to do that. I don't know. I assumed it would be fine. I could also ask it like, hey, is there anything else that we could do to refactor to make it more human readable? But I don't feel like it would know because it's not a human. And it's trained on garbage repos. Okay, so let's go into our main, wherever that is. Hold on here, our main. And I usually just have a sense of like, is this readable, right? and it's still YIC. It's really, really long. So there's still some stuff in here that needs to be refactored. We'll go over to here. So say coverage report. Coverage report should be its own file in lib called coverage report. Okay. The other thing is like the data. So Right now we have hard-coded data. Make a data folder and store data artifacts and load them into the app. Okay, that's another thing. I really just like the logging. And we have trace append. It's still very, very verbose. And there are still things, it's like, I'm noticing here, like, there are templates for... content for messages that should really be templated files that take variables and load it in. Maybe, okay, like, is that a prompt? I mean, like, we have this. It's technically our prompt. So technically, they are prompts, are prompts for content. And so prompts, so move them to prompts folders. Okay, so there's that there's a lot of those. Okay, and so we'll go back here. We'll save the file All the way down to here There are new tasks in the refactor MD Okay, and so we're gonna have it go off and do those tasks And so we'll give it a moment there I'm gonna pause and I mean I really should tell it to check them off I didn't tell it to do that But we'll come back and take a look and then we'll ask it to do a general refactor I'm still like I really don't like this like we see how we like double lines and stuff like that I don't need that kind of level logging, but I would have to explain to it why that's an issue. Yeah, it's still just making them empty files. It's not marking them whether they're templates or not, but we'll just treat them as templates. And here we're getting a lot more in here. So that's better. But I just know what good code looks like. And I know that's not good code. But there's only so much you can do with Python. Certain languages have the ability to have better readability like Ruby's really really good at that I'd love to port this over to Ruby I just didn't check if the agent SDK is available I don't think it is available in Ruby I just think that this anthropic one is and so if the agent SDK was in Ruby I would absolutely be using it over on the Python one as I really do not like Python code and it just you just can't get it to be extremely human readable unfortunately we were all kind of using it because of the way the industry is as they've adopted it, not because it's good, just because of mass adoption in the data science and stuff like that. So it just became de facto. Oh, look, it's already done. And so we have that there. And so we will, again, look at the main file. I'm just going to close it and reopen it. Sometimes it doesn't always show me right away. It didn't checkbox these off. I really should tell it to checkbox them off when it does that. So we'll go ahead and save that. We'll go back over to our main pie, and we'll scroll up. And again, I'm looking at this for refactorability. And I would say, like, they haven't done a good job with logging. So I'd say you haven't done a good job refactoring the logging, right? So, for example, we have log info partition, like partition. You should be making helper functions. So these logs. usually like log partition or you know like log right log warn and they will add the you know tags the other thing is you have superfluous logging that is great for human readability but we want to focus on logging good for for logs and We should be outputting Logs to a log folder relative to the folder of this agent Okay, and so that's one thing that's really bothering me. I really hate constants So that'll be another thing that we fix here in just a moment But again, we're just trying to get this to be in shape Did also move this out of here like what's this big thing like why is the tool used so large here? Okay And while that is thinking let's go review our other parts of code Okay, I mean this is fine I think I wouldn't mind if we had like this is a big JSON object But I wouldn't mind if we had a shorter syntax for this I don't want to do that right now because it's totally possible that we will be able to do that in agent SDK where it probably already has like a shorthand to make that code smaller. And so I don't want to muck with that. And we'll look at the partition here. Really hate those constants. And also I really just like how it's loading in the prompt template. So there should be a way to manage that. Like look at all this logger logic. Oh no, that's the logger file. Yeah, here now we're starting to get those things that I was asking for. That's good. um okay the other thing that that's and i mean we don't need to do this but like technically you know we have all these sub agents they're calling create we could technically delegate them out to different models if we needed to um or we could even say like hey can you try to choose the best model as it's working through there but yeah i guess the next thing on my task is to fix that um like i'm not updating the refactor we could keep updating that as a means to keep keep track of stuff which i just don't care um and so Yeah, I want to fix those constants. And I want to get something that loads in the templates. I'm just going to take a look here at our usage. 9% doing okay over here. Okay, and so now that is fixed up. We'll take a look here. Again, I'm looking at my main, seeing if it's shorter. Yeah, it's looking. Yeah, this is way less messier. I don't like using constants. E.g. like this is a var. please don't use these in the folder for the coordinator refactor fix the code okay so that's something i really dislike and so we will get that improvement there this this to me is like there's a big issue with the loop so i feel that we need to give it a better instructions on like how to better refactor the loop i mean it's using just a big elf itself block there might be some kind of uh state flow machine or something that could improve that loop as I'm not happy about it before we do that I want to fix the template reading and loading of files and so there it's just making basic changes for names right there so those are getting changed good and it'll be done here in probably just a moment yeah it's just updating the main.py and then we will have those fixed come on come on hurry up hurry up and also like loading these templates and populating them probably needs to be its own thing yeah great thanks Okay, another thing is loading files and templates where you inject variables. You can make a new template file in the lib directory and this should refactor having, you know, large load code eg like this okay and so that's another thing that's kind of bothering me so we will get that cleaned up as well um there's other things like this like see how this is like something's happening here so that should be refactored out into a function like everything here like just the units of code is is just not explainable so the run coordinator definitely needs to be broken up into tons of functions every single of these if else blocks should be functions um And I would probably prefer stateless classes. I really prefer stateless classes as that makes it really easy to track inputs and outputs of stuff. And Python's pretty good for that because of the way it defines these label tags. I can't remember what they call the name properties. And so that will be good. We're making a lot of changes here. So there's a high chance this might not work, but that's fine. We can always work through that. It's fine. You're fine, Claude. You're fine. okay and so now that's loaded i'm not sure if uh that actually changed we'll go back over to here and so with that now when it needs it i feel yeah like load prompt exactly like this um so yeah the big problem still is the run coordinator so the run coordinator file is giant uh we should refactor um into a stateless a stateless function and all the parts of code should be chunked into functions. So the functions basically act as documentation. For example, the contents of if blocks are really should be function calls, right? we'll go ahead and we'll see if it understands what I'm trying to say but like when you write good code you know like this would be a function this would be a function this would be a function whatever this is write these if blocks and we'll see if it understands what I'm talking about but I feel like that's a really important component in fact this run coordinator could also be in the lib directory And we might suggest it to move that in a moment. But right now, I want to see if it could even handle what I'm asking it to do. It might not understand. Because if I don't have like an example, it's just not going to know what I'm trying to ask for. Again, checking my coverage here. We're at 11% usage. Some folks were suggesting that like when it's high peak usage, it depends on like if you overlap with European time. I'm not obviously in Europe. And so they said like just try a bit later when everyone's sleeping. And it's way later. uh you know it would be maybe uh off peak usage time but anyway we'll just wait here and see what happens okay it's back with functions and again we could call plan to ask it to do this before but i don't really care that much so we have plan partitions validate index partitions run i've seen something with partition information almost wondering if that should be really part of the partitions pie right um log coordinator reasoning handle this information and so Those partition ones, there's three with partition ones, right? And so we'll go over to here. I mean, like, sure, you could do it that way. That's not what I asked for. I need to verify this. So I got to go over here. Like, what does a stateless class look like in Python? Can it be a class with static methods? Okay, because that's what I was asking for. Yeah, okay. So look, it did not do what I wanted. I mean close, so... Okay, we'll go all the way down here. You know, I wanted a stateless class. That is a class with static functions. Okay, right? So you didn't... And I noticed some of the functions were handling partition logic. Is that something that should really... be part of the partitions pie, right? So that's something I'm noticing as we are shaping that code, okay? And so we're going to give that a moment to shuffle those things around. Now, is it thinking about that or is it just shoving things over there? Index by agent for partitions, sure. Build initial messages. Again, is that for partition? Is it actually asking that question? Does it belong over there or is it just that it's handling partition data? because it moved it and didn't actually question whether it goes there or not. But anyway, we'll go over to here and we'll take a look of what's changed. What's it still working? So now this is looking a lot better because now we can see what is going in, what's going out, right? Okay. And so we have all these steps. So we have call. So create a message. Log reasoning. Again, like it does this logging stuff along with the logger handle screening agent Handle evaluation coverage handle files handle submit final process tool calls Run did they put these at the bottom? They did sometimes people put these at the top or sometimes they put the bottom but like the one that obviously is the big one is this one here And so the idea is that we should be able to easily see what it's doing. So we have generate partitions partitions validate overlap index agents right and so this should be self-documenting as you read it we go down here we call the coordinator we do the log reasoning why are these functions are these just loose functions they are well no they're part of the partition and so i would go over to here and i would say you know give the um give the partitions pi the partitions pi like all of our lib directory now i'll say Our partitionsPy should be a stateless class. So a class with static functions. Please update. And that's just the way I prefer it. Okay. I like to group them into domain. I don't like having loose functions where we don't know where they're coming from and who respects them or owns them. People in the data space are very much used to just randomly importing a bunch of stuff. So they... have a less sensitivity to that kind of thing but to me as a very professional developer i want to see where that stuff is coming from we still have some of our if else stuff here and notice in here like these again these should be functions right all they're all they're doing is calling these things but still i want these as functions if there's an if else statement in here especially in our main loop that's what it should be we have a range of one to thirty ones that's kind of defining how many steps it can take I would rather that uplifted as a variable, but we're not going to go too crazy on this. I just want to get it in enough shape here and mostly just to show you like what good code looks like and what you should be doing before you move on stuff. You might say, well, Andrew, why? Like this is more work to read. Yeah, but if you want to write test code for this, then you have an input and an output and you know exactly what to mock going in there and out of there. The only thing that I would also change is like if these are complex. objects I'd want them to be dumber and only pass in really dumb data so that we could mock it a lot easier and this is pain points if you've spent a lot of time writing a code and you might say well you know the AI can write the test code for us but that doesn't make it good tests or reliable tests and and you'll only know that by uh writing that stuff but we'll go over here we'll take a look at our partitions and so that is fine there's still lots of little improvements to be made like I'm looking like why is that like that I don't like hard-coded stuff like that um there's just a bunch of little things but i'll just say we'll move the coordinator over so uh the coordinator can be in its uh clear class should be in a its own file in the lib directory okay and we'll move that over there that'll be the last thing we do here um and then what we will do is we'll just run it make sure it still works and then we'll call this you know done-ish right but again you know if this was something that i would want to put in production i would take the time and fine-tune it i could take the time and fine-tune it and and because that's about getting um uh the word i'm looking for is um technical ownership, right? That you have ownership of the code and you know exactly what it's doing. And when you shape it like that, then you have a better sense of it. So now the coordinator is over there. I want to just run it to make sure it works. So I'm going to cd into the coordinator refactor. And we're going to go ahead and run python.main.py. Okay, so we're going to run that. And we will take a look and hopefully it still works. There we go. I wonder if it's going to make the log file. We do get our logs. Excellent. coordinator.log okay and see that's what i meant when i said i wanted to be nice and uh and whatever i might even suggest like i'd probably prefer to log out json structure because then we could parse that information um yeah i think that's what i would prefer especially like if you're data driven you have json ldata as logs it's super super useful um so instead of having like coordinator delegate um i would probably just have json objects and then i could parse it and ingest it into something else but again these are little tricks that you learn building applications of all kinds. But the point is that it is running. We just want to see it to completion and then we will call this done and then we will move on because the next section of stuff we are looking at is stuff that I feel like Agent SDK is going to be very useful for. They'll all have to decide on that. And so it did run. worked great the only thing that I'd probably ask it to do which it's not doing right now is I would have it dump the coverage report into its own file so that'll be the last thing that we do here okay and so I'm going to go here because that would actually make it useful right so I'm going to go and just say you know for my coordinator refactor it currently generates it generates a coverage report in the logs but it really should be outputting a report timestamped in a reports directory and formatted nicely for a human to read, right? And so that's the last thing I would absolutely say we need to do. I just realized that that's a little bit gross on how it currently is. And we never looked at our data, but yeah, we have our job posting and stuff. We could enrich these later, but they're fine. It's really next to new data here. We could have made a research that would go and grab job postings and make it for us. Not that anyone really should care about job postings anymore because agents are just going at it, but we'll wait for this to finish. Okay. And then we might run this one more time. Okay. There we go. And so it says it's there. The other thing is that I don't think we were logging usage. So that would be nice to be able to log that information out. But again, these might be things we get for free when we use the agent SDK. So I'm not exactly sure. And so now that it's done, I'm going to go ahead and run this one more time. Clear. And I have no idea how many credits I'm burning. Like, again, I haven't hit my... i have like five dollars or whatever i haven't hit it yet and bacon's not gonna get mad if i load up another five dollars so so far it is not a pain problem people don't know bako is the other andrew andrew b i'm andrew b and so we call him bako so it's not confusing he's definitely a real person he's not um an agent or is he we don't know no one ever sees him uh so we're gonna run this again i'm gonna pause here and then i just want to confirm the reports are there but again you can just see my thoughts of like what would be good to do okay We still have the coverage report being logged here, which I don't like, but that's fine. As long as we got a, I didn't, we didn't tell it to not do that there, but we'll go here. And then here is our report. We can go ahead and view it over like this. And so there is our final coverage assessment. I really don't like how long it's written this stuff. Like if you were a human, would you want to read this much information? Probably not. Or you'd want it summarized in a different way, but we never gave it a coverage report template. So. that's fine we will consider this done we'll say git add all git commit refactor but that wasn't bad for a quick refactor still lots of work to be done there right um i'll see you in the next one ciao ciao hey folks it's Andrew we are back and it's time for us to port our coordinator application over to agent SDK and the reason why is that we're going to be getting into um uh specific agent SDK um arguments and if we want to know how they work we need to have an example over there and I think we should just continue this project forward and I think it's not a bad idea so what we are going to do is we're going to call this port to to agent SDK okay and so what I'm going to do here is I'm going to grab the contents of all this not the logs we don't need the logs or the reports based on for my agent that uses directly the anthropic sdk to use clod agent sdk for this folder port sdk and so we're going to ask it to go ahead and do that that's a big thing again we probably should have put it into plan mode and ask it what it can do but i'm just going to go off to the races and do that and if it works we will explore it and um we'll have time to look at the code base quite a bit as we walk through other features okay all right let's take a look here and see what we have so we have the run updated i'm not sure why it did that it's not really that big of a deal uh we've removed the async anthropic and coordinator these are now internal to the coordinator sure it has a complete rewrite i was expecting that that i assume that would be the largest rewrite for us And I guess all those are unchanged. That's really interesting. And then we need to do a update here. I mean, you know, can you make the requirements.txt for me? Because that's what it should have done. But we never copied it from a prior one. That's probably why. Yeah, we didn't. So let's go take a look at the major changes. So we'll look at the main.py. And... Here we can see async anthropic. Oh, so there's where it's different default AIO, a client. That's why there was a change there. This is the new one, right? There we go. Okay. And so this more or less looks the same, but we'll go into our coordinator directory here and let's see if we can make the difference here. Okay. So I'm going to do a scroll up here. What I might do. Just so that we can really clearly see the difference, we might refactor a smaller one because it's very hard to see the changes. They don't even show us the code changes here, right? So what I'm going to do, I'm going to make another repo. We have, make another folder here. Let's see, port to agent SDK small. And the reason I want to do that again is to really clearly see the difference. And so I'm trying to think of one that we were doing before, like narrow task decomposition. Yeah, where we have this one. This one's a lot simpler, right? And we actually might want to go one step before that where we are using tool use. Could be decision-making, model-driven, right? So this one here is a very simple one with multiple tools. So what we'll do is we'll copy this. over here. I'm going to go into this directory just so we can clearly see the difference. And then also maybe just have another one that we can work on. Though I don't really like this use case per se. Okay. And so I'm going to go and say, okay, great. Can we convert the code for port to agent SDK small over to agent SDK? And again, we haven't tested if these actually work. And then the functions are probably defined a particular way. See, this whole big thing is probably gone. Yep. And so we have decorators on top of our functions making this code a lot smaller. OK. The call is a bit different. So that's one thing. And we are creating the SDK MCP server to pass the tools over. So that is another thing that's changing. OK. I mean, we have new modes and we're sending our MCP server with our tooling in it. Okay. So basically, we basically have an internal MCP. That's really interesting to make that super, super easy. And this call is a little bit different. So basically, the big thing that we're seeing is that tool use. So let's go back to our larger refactor. And I want to take a look at our tools. And so that tool, Jason, do we even need that anymore? Does that even make any sense? So what we'll do is go back over here, because now we know it was refactored, right? And we'll say, do we even need the tools JSON anymore? And shouldn't we be using the decorator for port to agent SDK based for clawed agent SDK? And I imagine that you can probably pass in that JSON tools because it's doing it. No, we don't know if it actually works or not. If we go here, tools.json, like I don't see it loaded in here. Maybe it's getting loaded in the main. It is refactoring probably right now, so we wouldn't even know. But we'll see what it says here. Because we do have tool right here, right? So it is, it's right here. So maybe it just has to delete it out. But if the tool's here, then why isn't that defined? Or does it have to sit in the same place, right? So we have this one here. Is this just a repeat? Okay, and like look at all this inline stuff, eh? Object, maybe pass, rationals, key strengths. That's the structure that we actually wanted from before. And so here, all three tool decorators are now using the simple param. Okay, but like does the coordinator still have them in here? Do we have to have the tools in the coordinator.py or can they live? in the tools directory as separate functions or it doesn't work because tight coupling of the decorator which is this part here it might be the reason why they can't do that and i mean i hope he knows what last directory we're in uh because we have more than one but uh we'll ask that question and you know this is what i'm trying to figure out what's it says here so The key insight tool to decorate runs at call time, not import time. So you can apply it inside the factory function to capture state via normal closures. Okay, speak English to me here. Can we move it or not? Coordinary class, tools, screening agent. Look, I'm trying to keep my stuff lean here, folks. Did it move it out? Did you even tell me that it moved it out? Okay. So here, coordinator state, make coordinator. So it did move it out from tools. I don't like how they're make coordinator tools. Okay. And then we have coordinator state. All right. Okay. I see. So they have a state file separately for it. I mean, state wouldn't belong in tools now, would it? So that doesn't make any sense. Unless it's coming from that file. Oh, maybe it's part of it. That's why. Okay. And so we go over to here and we have make coordinator tools. Oh, and they do have it in here. Okay, so they were able to move it out. And so here we have our multiple tools. Okay, and so to me, that's what I would like it to be. So I'm going to go ahead, and we're going to stop this, and we're going to cd into the port to agent SDK. And I just want to make sure that this still works. So we'll go ahead and say main.python. We got main, or main. Python, I got it backwards. Python main.python. Whatever, whoops. Okay. And I just want to make sure that it still runs because we've changed a lot of code or at least with large file to another framework. And it's logging. We'll pause here and see the end result, but I'm pretty certain that it's going to work. Okay, so it ran without issue and we are in good shape. Yeah, so we are set up and the question will be like, you know Do we use this to test out all the agent SDK stuff or do we come back to this project? We will see but at least we made it to this point and I think the key takeaway here is the fact that the tool use call got easier And it's setting up an MCP server. Okay, so literally it's an internal NPC server And so clearly Anthropic is obviously making that a priority tool. But anyway There we go, and we will move on from that. Okay, ciao, ciao.

Multi-agent systems get messy fast unless the orchestration layer is designed well.

This video is a focused deep dive into coordinator patterns in Claude Architect, including task decomposition, dynamic selection, research partitioning, refinement loops, observability, and porting the coordinator flow into the Agent SDK.

In this video:

- Hub and spoke architecture
- Coordinator design patterns
- Decomposition and dynamic routing
- Refinement and observability
- Coordinator refactoring
- Agent SDK porting

Full Claude Architect playlist:
https://www.youtube.com/playlist?list=PLBfufR7vyJJ71nHDB7kwcaih1O6sEowns

Full course on ExamPro:
https://www.exampro.co/cca-f

The full course also covers:

- SDK and environment setup
- Agentic loop foundations
- Execution control
- Session management
- Tool usage and MCP
- CI/CD and automation
- Scenario-based agent patterns

#ClaudeArchitect #Anthropic #MultiAgent #AIAgents #ClaudeCode #ExamPro