Claude Agent SDK [Full Workshop] — Thariq Shihipar, Anthropic

Okay, yeah, thanks for joining me. I'm still on the West Coast time, so it feels like I'm doing this at like 7 a.m. So, yeah, but I'm glad to talk to you about the Clawed Agent SDK. So, yeah, I think this is going to be a rough agenda, but we're going to talk about what is the Clawed Agent SDK, why use it? There's so many other agent frameworks. What is an agent? What is the agent framework? How do you design an agent using the agent SDK or just in general? And then I'm going to do some live coding, or Claude is going to do some live coding on prototyping an agent, and I've got some starter code. But yeah, the whole goal of this is like, we've got two hours, we're going to be super collaborative, ask questions. This is also going to be not like a super... canned demo in the sense that like we're going to be like thinking through things live, you know, I'm not going to have all the answers right away. And I think that'll be a good way of like building an agent loop, I think is like really very much like kind of an art or intuition. So, um, but yeah, before we get started, just curious, a show of hands, like how many people have heard of the cloud agent SDK or, okay, great. Cool. How many have like used it or tried it out? Okay, awesome. Okay, so pretty good show of hands. Yeah, so I'll just get started on the overview on agents. I think that this is something that people have seen before, but I think it's still taking some time to really sink in how AI features are evolving. So I think when GPT-3 came out, it was really about like single LLM feature right you're like oh like hey can you categorize this like return a response in one of these categories um and then we've got more like workflow like things right hey like can you like make this email and label it or like hey here's my code base like index for your rag can you give me like the next completion or the next um the next file to edit right and so that's what we call like a workflow where you're very like structure, you're like, hey, given this code, give me code back out. And now we're getting to agents. And the canonical agent is called code. Cloud code is a tool where you don't really tell it, we don't restrict what it can do, really. You're just talking to it in text, and it will take a really wide variety of actions. And so agents build their own context. decide their own trajectories are working very, very autonomously, right? And so, yeah, and I think as the future goes on, agents will get more and more economists, and we, I think it's like we're kind of at a break point where we can start to build these agents. They're not perfect, but it's definitely the right time to get started. So, yeah, Cloud Code, I'm sure many of you have tried or used. It is, yeah, I think the first true agent, right? Like the first time where I saw an AI working for like 10, 20, 30 minutes, right? So yeah, it's a coding agent. And the Cloud Agent SDK is actually built on top of Cloud Code. And the reason we did that is because basically we found that when we were building agents at Anthropic, we kept rebuilding the same parts over and over again. And so to give you a sense of what that looks like, of course, they're the models to start, right? And then in the harness, you've got tools, right? And that's sort of the first obvious step. Let's add some tools to this harness. And later on, we'll give an example of trying to build your own harness from scratch too and what that looks like. how challenging it can be, but tools are not just like your own custom tools, they might be tools to track with your file system, like with plot code. Did the volume just go up? Or were they not holding it close enough? Okay, start something not going to happen. Anyways, you've got tools, tools you run in a loop, and then you have the prompts, right? Like the core agent prompts, the prompts for the script, things like that. And then finally you have the file system. or not finally, but you have the file system. The file system is a way of context engineering that we'll talk more about later. And I think one of the key insights we have through Cloud Code was thinking a lot more through context, not just a prompt, it's also the tool, the files, the scripts that it can use. And then there are skills, which we've rolled out recently, and we can talk more about skills if that's interesting to you guys as well. And then, yeah, things like sub-agents, web search, research, compacting, hooks, memory. There are all these other things around the harness as well. And it ends up being quite a lot. So the Cloud Agent SDK is all of these things packaged up for you to use. And yeah, you have your application. So I think to give you a sense of To give you a sense of maybe why the Cloud Agent SDK is, yeah, people are already building agents on the SDK. A lot of software agents, software reliability, security, including triaging, bug finding, site and dashboard builders, these are extremely popular. If you're using it, you should absolutely use the SDK. MS Office agents, if you're doing any sort of office work, tons of examples there. Got some legal, finance, healthcare ones. So yeah, there were tons of people building on top of it. I want to, but yeah, okay. So why the Cloud Agent SDK? Why did we do it this way? Why did we build it on top of Cloud Code? we realized basically that as soon as we put Cloud Code out, yeah, the engineers started using it, but then the finance people started using it and the data science people started using it and the marketing people started using it. And yeah, I think it just like, we just realized that people were using Cloud Code for non-coding tasks. And we felt, and as we were building non-coding agents, we kept coming back to it, right? And so it's a like, And we'll go more into why that just works, why we could use Cloud Code for non-coding TAV. Spoiler alert, it's like the bash tool. But yeah, it was something that we saw as an emergent pattern that we want to use, and we built our agents on top of it, right? And these are lessons that we've learned from deploying Cloud Code that we've sort of baked in. So tool use errors or compacting or things like that, stuff that is like very, can take a lot of scale to find what are the best practices. We sort of baked into the Cloud Agent SDK. As a result, we have a lot of strong opinions on the best way to build agents. I think the Cloud Agent SDK is quite opinionated. I'll talk over some of these opinions and why we chose them, right? But yeah, one of the big opinions of the Bash tool is the most powerful agent tool. So, okay, what are like, what I would describe as the anthropic way to build agents, right? And I'm not saying that you can only build agents using the API this way, right? But this is like, if you're using our opinionated stack on the agent SDK, what is it, right? So roughly Unix primitives, like the bash and file system. And, you know, we're going to go over like prototyping an agent using cloud code. And my goal is really to sort of show you what that looks like in real time. right like why is bash useful why is the file system useful why not just use tools um yeah agents uh i mean you can also make workflows i'll talk about that a bit later the agents build their own contacts um thinking about code generation for non-coding um like we use code gen to generate docs create the web like do data analysis take unstructured action so um there's a lot of like uh this can be pretty counterintuitive to some people and again in the like prototyping session we'll go over how to use code generation for non-coding agents um and yeah every agent has a container or is hosted locally because this is plot code uh it needs a file system it needs bash it needs to be able to operate on it and so it's a very very different architecture i'm not planning to talk too much about the architecture today but we can at the end if that's what people are interested in in or sorry by architecture I mean hosting architecture like how do you host an agent and like what are best practices there have you talk about that at the end yeah so well let me pause there because I feel like I've covered a lot already any questions so far on the agent SDK agents yeah like what you get from it can you explain what code generation for non-coding means exactly yeah This is, like, basically, when you ask Cloud Code to do a task, right? Like, let's say that you ask it to find the weather in San Francisco and, like, you know, tell me what I should wear or something, right? Like, what it might do is it might start writing a script to fetch a weather API. right and then start like maybe it wants it to be reusable like maybe you want to do this pretty often right so it might fetch the weather api and then get the like maybe even get your location dynamically right based on your ip address and then it will like um you know check the weather and then maybe like call out to like a sub-agent to give you recommendations maybe there's an api for your closet or wardrobe right it's like so that's an example. I think that like it's kind of for any single example we can talk over how you might use code gen. A lot of it is like composing API is like the high level way to think about it. Yeah. Workflow versus agent, like for repetitive task or you know like a process, a business process that is always the same, do you will still prefer to build an agent versus a fully deterministic workflow? Yeah, so we do have Oh, sure, yeah. So the question was about workflows versus agents, and would you still use a Cloud Agent SDK for workflows. Is that right? Yes. And so, I mean, we just sort of tell you what we do internally, basically. And what we do internally is we've done a lot of GitHub automations and Slack automations built on the Cloud Agent SDK. So we have a bot that triages issues when it comes in. That's a pretty workflow-like thing. But we've still found that in order to try out issues, we wanted to be able to clone the code base and sometimes spin up a Docker container and test it and things like that. And so it still ends up being like a very, like there's a lot of steps in the middle that need to be quite free flowing. And then you give structured output at the end. So yes. All right, we'll take one more question and then keep going. So yeah, in the blue. Yeah, so could you talk about security and guardrails? Like, if we're using Cloud Agent SDK and you lean towards using Bash as the all-powerful generic tool, is the onus on building the agent builder to make sure that you're preventing against common attack vectors? Or is that something that the model is doing? Yeah, so I think this is sort of like the Swiss chief. Oh yeah, so the question was permissions on the bash tool, right? Or like how do you think about permissions and guardrails? Like when you're giving the agent this much power over, you know, it's an argument on the computer, how do you make sure it's aligned, right? And so the way we think about this is what we call like the Swiss cheese defense, right? So like there is like on every layer some defenses and together we hope that it like blocks everything right so obviously on the model layer we do a lot of alignment there we actually just put out a really good paper on reward hacking super recommend you check that out um so like definitely i think claw models like we try and make them very very aligned right and uh so yeah there's the model alignment behavior then there is like the harness itself right and so we have a lot of like permissioning and prompting um and uh We do an ASTPath parser on the bash tool, for example, so we know fairly reliably what the bash tool is actually doing, and definitely not something you want to build yourself. And then finally, the last layer is sandboxing. So let's say that someone has maliciously taken over your agent. What can it actually do? We've included a sandbox where you can sandbox network request and sandbox file system operations outside of the file system. And so, yeah, ultimately that's what they call the lethal trifecta, right, is the ability to execute code in an environment, change the file system, and exfiltrate the code, right. I think I'm getting the lethal trifecta a little bit wrong there, but the idea is basically if they can exfiltrate your information back out, right. that's like, they still need to be able to extract information. And so if you sandbox the network, that's a good way of doing it. If you're hosting on a sandbox container like Cloudflare, modal, or you know, E2B, Daytona, like all of these, like sandbox providers, they've also done like some level of security there, right? Like you're not hosting it on your personal computer or on a computer with like your product secrets or something. So yeah, lots of different layers there. And yeah, we can talk more about hosting in depth. Okay, so I'm gonna talk a little bit about Bash is all you need. I think this is something that... This is my shtick. I'm just gonna keep talking about this until everyone agrees with me. I think this is something that we found adentropic. I think it is something I discovered once I got here. Bash is what makes talkbook so good. You guys have probably seen code mode or programmatic tool use, different ways of composing MCPs. Cloudflare has put out some blog posts on that. We put out some blog posts. The way I think about code mode or bash is that it was the first code mode. So the bash tool allows you to store the results of your tool calls to files, store memory, dynamically generate scripts and call them. Compose functionality like tail graph I let you use existing software like FM peg early process, right? So there's a lot of like interesting things and powerful things that the bash tool can do and like think about like again What made cloud code so good if you were designing an agent harness? Maybe what you would do is you'd have a search tool and the lint tool and execute tool right and like, you know and tools right like every time you thought of like a new use case you're like I need to have another tool now, right? Instead, now Cloud just uses grep, right? Or knows your package manager, so it runs like npm run test.ts or index.ts or whatever, right? Like it's lint, right? And it can find out how you lint, right? And it can run npm run lint. If you don't have a linter, it can be like, what if I install eslint for you, right? So this is like, like I said, the first programmatic tool calling, first code mode, right? Like you can... do a lot of different actions very, very generically. And so to talk about this a little bit in the context of non-coding agents, so let's say that we have an email agent and the user is like, okay, how much did I spend on ride sharing this week? It's got one tool call, or generally it's got the ability to search your inbox. And so it can run a query like, hey, a search, Uber or Lyft, right? And without Bash, it searches Uber or Lyft, it gets like 100 emails or something, and now it's just got to like think about it. You know what I mean? And I think like a good analogy is sort of like imagine if someone came to you with like a stack of papers and like, hey, how much did I spend on ride sharing this week? Can you like read through my emails? You know what I mean? Like that would be... you're really hard, right? Like you need very, very good precision and recall to do it. Or with bash, right? Like let's say there's a Gmail search script, right? It takes a query function and then you can start to save that query function to a file or pipe it. You can grab four prices, you know, you can then add them together. You can check your work too, right? Like you can say, okay, let me grab all my prices, store those. as like in a file with line numbers and then let me then be able to check afterwards like uh was this actually a price like what does each one correlate to right so there's a lot more like dynamic information you can do to check your work with the bash tool so this is like um just a simple example but like hopefully showing you sort of the power of like the composability of bash right so i'll pause there any questions on bash is all you need the bash tool any any I think I can make a little bit clearer depth. Do you have stats on how many people use YOLO? Stats on YOLO mode? We probably do. I mean, internally, we don't. But I think we just have a higher security posture. Yeah, I'm not sure. I can probably pull that. Any other questions on Bash? OK, cool. Yeah, just to give you some more examples, let's say that you had an email API and you wanted to go through, like tell me who emailed me this week, right? So you've got two APIs, you've got an inbox API and a contact API. This is like a way you can do it via bash, you can also do it via code gen. This is kind of like enough bash that it is code gen, right? Bash is essentially a code gen tool. And then, yeah, like let's say that you wanted to, you had a video meeting agent. You want to say like, find all the moments where the speaker says quarterly results in this earnings call. You can use F of Mpeg to slice up this video. You can use JQ to start analyzing the information afterwards. So yeah, lots of powerful ways to use Bash. So I'm going to talk a little bit about workflows and agents. You can do both. You can use build workflows and agents on the Agent SDK. Yeah, agents are like cloud codes. If you are building something where you want to talk to it in natural language and take action flexibly, then that's where you're building an agent. You have an agent that talks to your business data and you want to get insights or dashboards or answer questions or write code or something, that's an agent. And then a workflow is kind of like, we do a lot of GitHub actions, for example, so you define the inputs and outputs. very closely, right? So you're like, okay, take it a PR and give me a code review. And yeah, both of these you can use the agent SDK for. When building workflows, you can use structured outputs. We just released this. You can, yeah, Google agent SDK structured outputs. But yeah, so you can do both. I'm going to primarily be talking about agents right now. A lot of the things that you can learn from this are applicable to workflows as well. Yeah, we'll talk about this. Show of hands, how many people have designed an agent loop before? Okay, cool, okay, great, great. So yeah, I think the number one thing, the meta learning for designing an agent loop to me is just to read the transcripts over and over again. Every time you see the agent run, just read it and figure out like, hey, what is it doing? Why is it doing this? Can I help it out somehow? And we'll do some of that later. So we'll build an agent loop. But here is the three parts to an agent loop. First, it's gather context. Second is taking action. And the third is verifying to work. And this is not... the only way to build an agent but I think a pretty good way to think about it. Gathering context is like you know for clock code it's grabbing and finding the files needed right you know for an email agent it's like finding the relevant emails right and so these are all like pretty yeah like I think thinking about how it finds this context is very important and I think a lot of people sort of skip the step or like underthink it. This can be very, very important. And then taking action, how does it do its work? Does it have the right tools to do it, like code generation, bash? These are more flexible ways of taking action, right? And then verification is another really important step. And so basically what I'd say right now is if you're thinking about building an agent, think about can you verify its work? And if you can verify its work, it's a great like candidate for an agent. If you can't verify its work, like it's like you know coding you can verify by linting right and you can at least make sure it compiles so that's great. If you're doing let's say deep research for example it's actually a lot harder to verify your work. One way you can do it is by citing sources right so that's like a step in verification but obviously research is less verifiable than code in some ways right because like code has a compile step right you can also like execute it and see what it does right so I think thinking on, as we build agents, the ones that are closest to being very general are the ones with the verification step that is very strong. I think there was a question here. Yeah. So, when, where do you generate it? Yeah, I mean, you might. Oh yeah, sorry. The question was, when do you generate a plan before you run through it? Like in cloth code, you don't always generate a plan, but if you want to, you'd insert it between the gathering context and taking action step, right? And so plans sort of help the agent think through step by step, but they add some latency, right? And so there is like some trade-off there. But yeah, the agent SDK helps you do some planning as well. Can you make the agent create that to-do list? 100% sure that it will create that to-do list and run by it? Yeah, so the question was, will the agent create the to-do list? Yes. If you're using the NSPK, we have some to-do tools that come with it, and so it will maintain and check off to-do lists, and you can display them as you go. Any other questions about this right now? OK, cool. OK, so I'm going to quickly talk about how do you do this stuff. What are your tools for doing it? There are three things you can do. You have tools, bash, and cogeneration. Traditionally, I think a lot of people are only thinking about tools. Basically, one of the call to actions is just figuring out thinking about it more broadly. Tools are extremely structured and very, very reliable. If you want to have as fast an output as possible with minimal errors, minimal retries. tools are great. Cons, they're high context usage. If anyone's built an agent with like 50 or 100 tools, right, like they take up a lot of context and the model, it kind of gets a little bit confused, right? There's no like sort of discoverability of the tools and they're not composable, right? And I say tools in the sense of like, if you're using, you know, messages or completion API right now. that's how the tools work. Of course, there's code mode and programmatic tool calling, so you can sort of blend some of these. Then there's bash. So bash is very composable, like static scripts, low context usage. It can take a little bit more discovery time, because let's say that you have the Playwright MCP or something like that, or sorry, the Playwright CLI, the Playwright Bash tool. You can do playwright-help to figure out all the things you can do, but the agent needs to do that every time, right? So it needs to discover what it can do, which is kind of powerful. That helps take away some of the high context usage, but add some latency. There might be slightly lower call rates, you know, just because it has a little bit more time to... It needs to find the tools and what it can do, but this will definitely improve as it goes. And then finally, CodeGen. highly composable dynamic scripts, they take the longest to execute, right? So they need linking, possibly compilation, API design becomes like a very, very interesting step here, right? And I'll talk more about like best, like how to think about API design in an agent. But yeah, I think this is like how we, like the three tools you have. And so yeah, using tools, I think you still want some tools, but you wanna think about them As atomic actions, your agent usually needs to execute in sequence, and you need a lot of control over it. So for example, in Cloud Code, we don't use bash to write a file. We have a write file tool, because we want the user to be able to see the output and approve it. And we're not really composing write file with other things. It's a very atomic action. Sending an email is another example. Any sort of non-destructible or un-destructible reversible change is definitely like a tool is a good place for that. Then we've got bash, so for example there are like composable actions like searching a folder using GitHub, linting code and checking for errors or memory, and so yeah you can write files to memory and that can be your bash, like bash can be your memory system for example right so and then finally you've got code generation right so if you're trying to do this like highly dynamic very flexible logic composing APIs, like you're doing data analysis or deep research or reusing patterns. And so we'll talk more about code generation in a bit. Any questions so far about the SDK loop or tools versus bash versus code gen? Yeah. Yeah. I was going to ask, are you going to have any ready-made tools for altering results? I'm floating to a whole result like into the file system or... Like, let's say it goes into action and the context explodes. Is it like type the command that like, do everything else? Okay. Or otherwise just like long outputs believing in history? Sure, yeah. Like all the time just uploading them to files? Yeah, yeah. I think that's a good common practice. I think we... I remember seeing some PRs about this very recently on Cloud Code about handling very long outputs. And I don't know exactly. I think we are moving towards a place where more and more things are being just stored in the file system. And this is a good example. It's storing long outputs over time. I think generally prompting the agent to do this is a good way to think about it. Even if you have, I think something I just do always now is whenever I have a tool call, I save the results of the tool call to the file system so that you can search for us it and then have the tool call return the path of the result. Just because that helps it recheck its work. Yes? Yeah, so the question was about skills and like do we need skills to use bash better? Yeah, for context skills, maybe again... Skills, okay, yeah. Skills are basically a way of like, you know, allowing our agent to take longer complex tasks and like sort of load in things via context, right? For example, we have a bunch of docx skills, and these docx skills tell it how to do code generation to generate these files. I think overall skills are basically just a collection of files. They're also an example of being very file system or bash tool-pilled, because they're really just folders that your agent can CD into and read. What we found the skills are really good for is pretty repeatable instructions that need a lot of expertise in them. For example, we released a front-end design skill recently that I really, really like. It's really just a very detailed and good prompt on how to do front-end design, but it comes from our best... AI front-end engineer, you know what I mean? And he really put a lot of top thought and iteration to it. So that's one way of using skills. Yeah. Quick question. Yes. Sure. Should some stuff be related to Claude.md and some other stuff? So the question was about skill.md versus Claude.md and how to think about that, right? I will say all of these concepts are so new. You know, I mean, like even clock code is like released at like eight or nine months ago, right? Like, and so skills were released like two weeks ago. Like I like, I won't pretend to know all of the best practices for everything, right? I think generally skills are a form of progressive context disclosure. And that's sort of a pattern that we've talked about a bunch, right? Like with like bash and, you know, like. preferring that over like, you know, purely like normal tool calls. It's like, it's a way of like the agent being like, okay, I need to do this. Let me find out how to do this and then let me read in the skill at MD, right? So you ask it to make a docx file and then it like CDs into the directory, reads how to do it, write some scripts and keeps going. So yeah, I think like there's still some intuition to build around like what exactly you like define as a skill and how you split it out. um but uh yeah i think uh yeah lots of best practices to learn there still um yeah so the question was are skills ultimately part of the model um are they a way to bridge the gap i missed barry's talk at barry mentioned talk yesterday but uh yeah i think roughly the idea is that the model will get better and better at doing a wide variety of tasks and skills are the best way to give it out of distribution tasks right um but i i would broadly say that like it's really really hard especially like you know if you're like uh not at a lab to like tell where the models are going exactly um my general rule of thumb is I try and rethink or rewrite my agent code every six months just because I'm like, things have probably changed enough that I've baked in some assumptions here. And so I think that our agent SDK is built to, as much as possible, advance with capabilities. The bash tool will get better and better. We're building on top of cloud code. So as cloud code evolves, you'll get those wins out of the gate. But at the same time, things are so different now than they were a year ago in terms of AI engineering. And I think a general best practice to me is sort of like, hey, we can write code 10 times faster. We should throw out code 10 times faster as well. And I think thinking about not so hedging your bets on where is the future right now, but what can we do today that really works? like let's get market share today and not be afraid to throw out code later. If you're a startup, this is arguably your largest advantage that you have over competitors. They're like, you know, larger companies have like six month incubation cycles. And so they're always like stuck in the past of like the agent capabilities, right? And so your advantage is that you can like be like, hey, the capabilities are here right now. Let me build something that uses this right now. Right. So, Yeah, any other questions on, we're talking about skills in bash, okay, it seems like there are a lot of skill questions, so yeah, I think at the back, someone, you might have to shout. Yeah, so why would you use a skill versus an API? Yeah, so the question was why use a skill versus an API? Good question, I think that like, These are all forms of progressive disclosure basically to the agent to figure out what it needs to do. And I'll go over examples of like, you just have an API in our prototyping session. It's totally use case dependent. I don't think there's a general rule. I think it's like, read the transcript and see what your agent wants. If your agent always wants like things about the API better is like a API.ts file or something or API.py file, do that. You know, that's great. Like I think skills are like, oh, like sort of an introduction into like thinking about the file system as a way of storing context, right? And they're a great abstraction. But there are many ways to use the file system. And I should say that like something about skills is that like you need the bash tool, you need a... virtual file system, things like that. So the agent SDK is like basically the only way to really use skills to like their full extent right now. So yeah, yeah back there. Yeah, the question was, can we expect a marketplace for skills? So yeah, ClockCode has a plugin marketplace that you can also use with the agent SDK. We're evolving that over time, you know, like it was like a very much a V0. And by marketplace, I'm not sure if people will be charging for this exactly. It's more just like a discovery system, I think. But yeah, that exists right now. You can do slash plugins in Cloud Code, and you can find some. So, yeah. What's your current thinking about when you're going to reach for the SDK to solve a problem? Yes, the question is when do I use the SDK to solve a problem? If I'm building an agent, basically. I think that, like, My overall belief is that for any agent, the bash tool gives you so much power and flexibility and using the file system gives you so much power and flexibility that you can always eat out performance gains over it. In the prototyping part of this talk, we're going to look at an example with only tools and an example with bash and the file system and compare those two. That's what I mean by that being bashful built. I just start from the agent SDK, and I think a lot of people at Anthropic have started doing that as well. Of course, I do want to say that there are lots of times where the agent SDK is kind of annoying because you've got this network sandbox container and you're like, I hate, I don't want to do this, I want to run on my browser locally, right? I totally get that. I think there is a real performance trade-off. The way I think about it is sort of like... React versus jQuery. When I was coming up, I was very into web dev and I was using jQuery and Backbone and then React came out and it was by Facebook and they're like, here's JSX, we just made this up and now there's a bundler. It's so annoying. But it generally makes the model or it makes web apps more powerful. And I think we're sort of like The agent is because you're like the reactive agent frameworks to me because it's like we build our own stuff on top of it so you know it's real and all the annoying parts of it are just like things where we're annoyed about it too but we're like it just works like you have like you gotta do this you know so yeah yeah okay more more skill questions I guess yeah right here oh sure the bash question great I love it you got custom internal like bash tools yeah How do you investigate and discover that? What do you know it has to become a tool that's cool? OK, the question is, if you have custom agent bash tools, how do you let the agent discover that? By custom bash tools, do you mean like bash scripts? Yeah, bash scripts, yeah. Yeah. Yeah, so I think, where is it? You just put it in the file system, and you tell it, like, hey, here is a script. You can call it. I'm generally thinking in the context of the Cloud Agent SDK, where it has the file system and the bash tools are tied together. This is kind of an anti-pattern I see sometimes where people are like, oh, we're going to host the bash tool in this virtualized place, and it's not going to interact with other parts of the agent loop. And that makes it hard, because if you've got a tool result that's saving a file, then your bash tool can't read it, unless it's all in one container. So does that answer your question? Yeah, kind of. So you're saying you would put it in like system prompt or something? Yeah, just put in system prompt being like, hey, you have access to this. I would sort of design all my CLI scripts to have a dash dash help or something so that the model can call that. And then it can progressively disclose every subcommand inside of the script. Yeah, . Yeah. So my question is when to reach for the agent SDK. So have you designed, or rather, would you recommend someone use the agent SDK to build a generic chat agent as compared to like oh you know i'm building an agent where you have some input and the agent goes and does some stuff and finally i care about the output as compared to let's say someone like are you using or do you foresee using the agent to build like the agent sdk to build like cloud the the app rather than cloud code uh yeah so the question is when do we reach for the agent sdk uh if it does um Like, would we use the Agent SDK to build Cloud.ai, which is a more traditional chatbot than Cloud Code? One, I think Cloud Code is like a very, like interface is not a traditional chatbot interface, but like the inputs and outputs are, right? Like you input code in, you get like, or you input text in, you get text out, and you take actions along the way. You might have seen that when we rolled out doc creation for Cloud.ai, now it has the ability to spin up a file system and create spreadsheets and PowerPoint files and things like that by generating code. And so that is like, we're in the midst of merging our agent loops and stuff like that, but broadly, Cloud.ai will... It's getting more and more, you see it with skills and the memory tool and stuff, more and more file system-filled. So we do think it's a broad thing that you can use just generally. And happy to talk through examples. Yeah, one more question, then we'll keep going. Yeah. Still trying to understand the rule of thumb on when to build a tool or use a tool, when to grab something with a script, or just let the agent go wild on the bash. I'll give you an example. Let's say I need to access a database from time to time. I can use an MCP. I can wrap it in a script. And I can just let the agent call an endpoint directly from Bash. Yeah, great question. Great question. So still trying to grok when to use tools versus Bash versus Cogen. And he gave an example like, OK, I have a database. I want the agent to be able to access it in some way. What should I do? Should I create a tool that queries the database in some way? Should I use the bash? Should I use CodeGen? These are all, these are three ways of doing it. I think that they are, like you could use any of them. And I think like part of it is like, I think unfortunately there's no like single best practice, right? This is like kind of a system design problem. But let's say that you want to access your bash, your database via a tool. You would do that if your database was very, very structured and you have to be very careful about like you're accessing user sensitive information or something like that and you're like, hey, I can only take in this input and I need to give this output and I have to mask everything else about the database from the agent. Obviously that sort of limits what the agent can do. It can't write a very dynamic query. If you're writing a full on SQL query, I would definitely use Bash or QoGen just because when the model is writing the SQL query, it can make mistakes. And the way it fixes its mistakes is by like linting or like running the file, looking at the output, seeing if there are errors and then iterating on it, right? And so I generally like, if I'm building an agent today, I'm giving it as much access to my database as possible. And then I'm like putting in guard risks, right? Like I'm probably limiting it's like write access in different ways. But probably what I would do is I would give it write access and put in specific rules and then give it feedback if it tries to do something it can't do. You know what I mean? And so I know this is kind of a hard problem, but I think this is the set of problems for us to solve. We built a bash tool parser, and that's a super annoying problem. But we need to solve that in order to let the agent work generally. And same thing with databases. Yes, it's quite hard to understand what is a query doing, but if you can solve that, you can let your agent work more generally over time. I think thinking about it flexibly as much as possible and keeping tools to be very, very atomic actions that you need a lot of guarantees around. A follow up on the same thing. How do you ensure that role-based access controls are taken care of? The question is, how do you ensure that the role-based access controls are taken care of? Usually, that's in how you provision your API key or your back-end service or something like that. I think that probably what I do is I create temporary API keys. sometimes people create proxies in between to insert the API keys if you're concerned about exfiltration of that but yeah I would create like API keys for your agents that are scoped in certain ways and so then on the back end you can sort of check it's like you know what it's trying to do and like if it's a an agent you can like give it different feedback so yeah all right one question anything you could tell us more about the the memory tool the internal memory tool I'm not trying to keep a secret. I don't know exactly. I haven't read the code. But I think it generally works on the file system. And so it's supposed to be a cloud agent SDK, or is it already a big one? I would say that we've had this question a bunch. I would just use the file system in the cloud agent SDK. I would just create a memories folder or something and tell it to write memories there. I don't know the exact implementation of the memory tool, but it does use the file system in that way. So yeah. All right, yeah, last question on this. Yeah. How you are managing, for the bash and the code, how you are managing the reusability? Suppose the same agent is all out to Microsoft users, and same code every time it is generating and every time it is executing. So how can we use the reusability? Yeah, that's a really good question. Yeah, let's say you have two agents interacting with two different people. The question is like, how do you think about reusability between agents or how do agents communicate, right? I think this is a thing to be discovered. I think there's a lot of best practices and system design to be done on like, because traditionally with web apps, you're serving one app to like a million people, right? And with agents, Like with Cloud Code, we serve a one-to-one container. When you use Cloud Code on the web, it's your container. And so there's not a lot of communication between containers. It's a very, very different paradigm. I'm not going to say that I know exactly the best system designed to do that. And I think there's lots of best practices on, OK, these agents are reusing work. How can we give them like cut? general scripts that combine together the work that they've done, how can we make them share it. I would generally think, this is sort of like a tangent on like agent communication frameworks, I would say that like we probably don't need like a whole, we don't, I think there's more of a personal opinion, I think like we probably don't need to reinvent like a new communication system, the agents are good at using the things that we have like HTTP requests and bash tools and API keys and named pipes and all of these things. And so probably the agents are just making HTTP requests back and forth from each other using HTTP server. There's a bunch of interesting work there. I've seen people make a virtual forum for their agents to communicate and they post topics. reply and stuff like that. Kind of cool, I think there's a lot of things to explore and discover there. Okay, I'm gonna keep going a little bit. How are we doing for time? Okay, it's got an hour left, I think. Okay, cool, so an example of designing an agent. This is not the prototyping session, but I think this will be a good way into it. Let's say we're making a spreadsheet agent. What is the best way to search a spreadsheet? What's the best way to execute code? Or what's the best way to take action in a spreadsheet? What is the best way to link a spreadsheet? These are all really interesting things to do. I'm going to do a figma and we can go over it. If someone could grab a water as well, that would be great. I could really use water, man. Yeah, okay. Thanks. Okay, so we're going to, yeah, let's talk through it. Or why don't you spend a couple minutes yourselves thinking about this question. You have a spreadsheet agent. You want it to be able to search, you want it to be able to gather context, take action, verify its work, how would you think about it? So just spend some time thinking through that, take some notes or something. A little bit of time to think about this. Does anyone want more time or just dive into it? Okay. What's the best way for an agent to search a spreadsheet? Realizing I have to type with one hand now. I should figure this out because I'm going to need to type later. Okay, searching a spreadsheet. Any ideas? How do you search a spreadsheet? What would you do? CSV. Okay, you've got a CSV. Okay, now your agent wants to search the CSV. What does it do? A grep sitage. Okay. What does the grep look like? Can you just look at all the headers? Looks at the headers, okay. Headers of all sheets. Okay, great, yeah. And let's say I'm looking for the revenue in 2024 or something. Now I've got my headers, like, I'm just going to pull up a spreadsheet, right? Let's say that the revenue is in, there's a revenue column, and then there's like a, so yeah, let's see. Okay, so yeah, let's say it's something like this, right? Like, how do I get revenue in 2026, right? So this is sort of like a tabular problem, right? Like, there is revenue here, and there's also 2026 here, right? So it's like a multi-dimensional step, right? We could look at the headers that will then give us, like, if you just pull this, you'll get 100, 200, 300, right? So we need a little bit more. Any other ideas? There's a bash tool for it, the awk, AWK, I think. Awk? Okay. Yeah, yeah, yeah. And what would it awk for? Well, it depends on what you're looking for. Yeah, yeah, yeah. That's a question, right? Like, what is the user looking for, right? They're probably looking for something like this, like revenue in 2026, right? Maybe use the APIs to use the Google tools to add all the numbers together. We look up something else. Yeah, so the idea is like use the API's, like use the Google API's to like look it up. That's great, but yeah, let's say we're working locally, we need to sort of design these API's, yeah? SQLite or .DB can query a CSV directly. Oh, interesting. Okay, yeah, I didn't know that. That's great. So yeah, you use SQLite to query a CSV. That's a great, like sort of creative way of thinking about API interfaces, right? Like if you can translate something into interface that the agent knows very well that's great right and so like if you have a data source if you can convert it into a sequel query then your agent really knows how to search sequel right so thinking about this transformation step is really really interesting it's a great way of like designing like an agentic searching phase so yeah yeah Because that's kind of what we're talking about here. Right tool for the right job. Yeah, is Claude smart enough to rank the right tool for their job? Yeah, if you prompt it. I think that's one of those things where, like, I don't know. Let's find out. Let's read the transcript. If it's not, how can you help it? Yeah, just sort of like, I think all of these things are like an intuition. It's kind of like riding a horse. Not that I've ever rode a horse, but I imagine just like running a horse. Yeah, like you're sort of giving these signals to the horse or calming it down. You're trying to understand how do you push it faster, you know what I mean? And sort of like, it's a very organic thing, right? I think we like to say that models are grown and not designed, right? So we're like sort of understanding their capabilities, yeah. Yeah, so that's another great pattern is like, okay, can you add metadata to a spreadsheet? So these are some questions that you might want to think about before, like when you're thinking about search is like what pre-processing can you do? to make the search better right and so one example is that you translate it into like a sequel format or something where you use something that can query it right that's like a translation step another step is like maybe you have a tool or um like a pre-processing step where another agent annotates the the spreadsheet and and like adds information so that the agent can then like search across that information better right so um yeah one more um i was just curious I mean, all those tools sound great, but why can't the agent just do what was suggested, read the header, and then just get the date? I feel like that should be pretty trivial for free tasks. Yeah, probably I should have prepared this in code. But yeah, I built a ton of spreadsheet agents before basically. Does that work? It's kind of hard to do. Yeah, so basically what I would think about is like, so we've got like, Okay, I, Sean, do you have suggestions on how it can, how it can code at the same time? Go ahead. Install voice to text on your computer. Oh, I see. Yeah, yeah, yeah. Have you worked at Whisperful or something? Stick the mic in your shirt. There's a microphone button on the back. There's a microphone button on the back. Stick the mic in your shirt. Oh, I just don't trust that stuff, man. Okay. Maybe I shouldn't be working in an AI lab. Okay, so let's see. Hold on, hold on. So one way to do it is like, you see in spreadsheets, right, like you can say here, you can design formulas, right? So like B3, 2, right? So this is a syntax, for example, that the agent's pretty familiar with, like B3 to B5, right? And so you can design an agentic search interface, which is like this, right? Like B3, B5 or something, right? So like your agentic search interface can take in a range, right? You can take in a range string, right? And these are things that like the agent knows pretty well, right? Like you can do SQL queries, right? The agent knows SQL queries pretty well, right? And like these, you can also do XML. Sorry, the font is so small. Okay. Yeah, you can also do XML. I'm not sure if you guys know, but like, XLX files are XML in the backend, right? And XML is very structured. You can do like an XML search query. And there are different libraries that can do that. So that's one example, right? It's like, how do you search and gather context? And I hope this sort of like illustrates to you that like gather context is really, really creative, right? Like, and like, there's so many iterations. And if you've just, if you've only tried one iteration, it's probably not enough, right? Like, think about like, as many different ways as you can, like, try these out, right? Like, try SQL, or try, try the search, try the grep and oc, and like all of these things. have a few tests that you're trying across different things and see what the agent likes and what it what it doesn't like it's going to be different for each case yeah yeah and you're relying on already who's doing that for model? Yeah, because the question is like, where does the knowledge come from? Is it the model? Is it like, what do I mean by the agent? Yeah, generally, I think what you're looking for is like, you have a problem, you want to make it as in distribution as possible for the agent, right? And so the agent knows a lot about a lot of different things. It knows a lot about, for example, finance, right? So if you ask it to make a DCF model, it knows what DCF is. right and you can if if you want to give them more information you can make a skill right but so it knows what dcf is it knows what sql is can it combine those things together right and so like uh ideally you want to like your your problem is going to be out of distribution in some way right like like there's some like information that's not on the internet or something that you have uh or something is somewhat unique to you and you want to try and like massage it to be as in distribution as possible And yes, it's very, very creative, I think. It's not a science to be very much like an art. So we've tried gathering context, then taking action. We can probably do a lot of the same stuff here that we've done before. We can do insert 2D array. If we've got a SQL interface, we can do a SQL query, we can edit XML. These are often very similar, like taking action and gathering context. You probably want a similar API back and forth. And then the last thing is verifying work. How do you think about that? Check for null pointers is one of the ways to do it. Any other ideas on verification? Sorry, I'm a bit confused. When you're using other SDKs to build Asian, I don't need to tell it how it should gather the context. I just give it the context and explain. Basically, I explain in plain English what it's meant to do. What I tend to do and you tell me if I'm wrong, I actually end up creating a separate agent for QA. Oh, interesting. To verify because I don't trust the agent to verify itself. But I'm just a bit confused about the level of detail I need to provide the agent in that example. Yeah, okay, so the question is about giving context to the agent versus having it gather its own context. You mentioned that you sometimes use a Q&A agent. Can I ask what domain you're building your agent in? In cybersecurity. OK, sure. Yeah. I think that I think I need to look into more specifics, but the Cloud Agent SDK is great for cybersecurity. And I would generally push people on, let the agent gather context as much as possible. you know like let it find its own work as much as possible um you're trying to give it the tools to find its own work the way i think about this is kind of like let's say they someone locked you in a room and they were they were like giving you tasks you know like so that's what your what your job was like a mr beast sort of like scenario right like you get five hundred thousand dollars if you stay in this room for six months um then like Like someone's giving you a message, what tools would you want to be able to do it, right? Like would you just want like a list of papers or like would you want a calculator or like a computer, right? Probably I would want a computer, right? I'd want Google, I'd want like all of these things, right? And so like I wouldn't want the person to send me like a stack of papers being like, hey, this is probably all the information you need. I'd rather just be like, hey, just give me a computer, give me the problem, let me search it and figure it out, right? And so that's how I think about agents as well. Like they need like, you know, they're stuck in a room. I need to give them tools. So if you can go back to the slide you have, to the graph you have. To the graph, like this, you can? Yeah, the stuff. So basically, that gathering context is basically, these are the tools I'm offering. Yeah, exactly. Yeah, I'm giving it like. Maybe an API for code generation. Maybe I'm giving a SQL tool. Maybe I'm giving a bash. These are all examples. So yeah. You have one more question? So for all the agents that you're having uncertainty, do they share the same context window? Interesting. So do agents share the context window? Interesting question is overall about how you manage context. I think, and I haven't talked about this too much yet, but sub-agents are like a very, very important way of managing context. I think that this is like, we're using more and more sub-agents inside of Cloud Code, and I would think about like doing sub-agents very generally. So like, what we might do for this spreadsheet agent is maybe we have a search sub-agent, right? So like, Sub-agents are great for when you need to do a lot of work and return an answer to the main agent. So for search, let's say the question is like, how do I find my revenue in 2026? Maybe you need to do a bunch of results. Maybe you need to like search the internet, maybe you need to search the spreadsheet, things like that. And there's a bunch of things that don't need to go into the context of the main agent. The main agent just needs to see the follow result, right? And so that's a great sub-agent task. I don't have a dedicated sub-agent side here, but like, yeah. They're very, very useful and I think a great way to think about things. Yeah, right there. And just to build on that question actually, for verification, for example, you can imagine doing that through a skilled or a sub-agent. You might even want to have an adversarial security example. It's a great one. You want to really go to town on it and not really have any sympathetic relationships with some work already done. I get it's a spectrum, but do you like? Are you saying, yes, you'd use a sub-agent here, you'd use a skill? How would you think about this? Yeah, definitely. So question on, like, do you sub-agents? I'm sure it'll work just to make sure. Oh, sure. OK, yeah, yeah. Thank you. Appreciate it. OK, yeah. Can you sub-agents for verification? Yes. I think this is a pattern. I think, like, ideally, the best form of verification is rule-based, right? You're like, is there, like, a null pointer or something? That's easy verification. It doesn't lint or compile. As many rules as you can try and insert them. And again, be creative. For example, in Cloud Code, if the agent tries to write to a file that we know it hasn't read yet, we haven't seen it enter the read cache, we throw it an error. We tell it, hey, you haven't read this file yet. Try reading it first. And that's an example of a deterministic tool. that we insert into the verification step and so as much as possible like anytime you are thinking about you know verification first step is like what can you do deterministically what like what like you know outputs can you do and again like when you're choosing which like types of agents to make the agents that have more deterministic rules are better you know like they just like like it just makes a lot of sense right so um Of course, as the models get better and better at reasoning, then you can have these sub-agents that check the work of the main agent. The main thing there is to avoid context pollution, so you probably wouldn't want to fork the context. You'd probably want to start a new context session and just be like, hey, you adversarily check the work of... This output was made by a junior analyst at McKinsey or something. They graduated from... like not a great school like your GPA like you know like like just like feed it a bunch of stuff and then tell it to critique it right like that's like one of the tools of the sub-agent right and so yeah the more you like yeah that's about to get better and better that sort of verification will become better as well but doing it deterministically it's like a great start you know question about the verify works yeah So let's say we found no pointers, it's probably easy to just say, OK, fix it. But let's say we could put the production and the client using it that's not us. And they somehow get into a spot where the whole spreadsheet is deleted. And so on what level do we need to bake in the ability to undo rules? Because let's say the QA agent returns that their spreadsheet is empty. Yeah. Not necessarily is able to undo. So what was your advice there? Yeah, so the question is, how do you think about state and undoing and redoing, being able to fix errors, basically? I think this is a really good question. And honestly, another sort of like, when you think about what are agents good at, or what problem domains are agents good at, how reversible is the work is like a really good intuition, right? So code is quite reversible. You can just like go back, you can undo the Git history. We come with like, you know, these atomic operations right out of the gate, right? Like I use Git constantly through cloud code. I don't type Git commands anymore, right? So that's like a really good example. A really bad example is computer use, you know? Because computer use is not reversible in state, right? Like let's say you go to like DoorDash.com and you add like the user wants you to order a coke and you add order a pepsi Now like you can't just go back and click on the coke You have to like go to the cart and you have to remove the pepsi right and so your mistake is like compounded this like You know this state and the state machine has gotten more complex And so whenever you're dealing with very, very complex state machines that you can't undo or redo of, it does become harder. And I think one of the questions for you as an engineer is, can you turn this into a reversible state machine, kind of like you said? Can you store a state between checkpoints such that the user can be like, oh, my spreadsheet is messed up right now. Just go back to the previous checkpoint. Potentially, even can the model go back to previous checkpoints? I think someone had this time travel tool that they were giving one of the coding agents, which was kind of cool, where you're like, it's like you can time travel back to a point before this happened, you know what I mean? It's kind of fun. I think all of these tools, some of them don't work that well yet, but we'll get there. Yeah, thinking about state and verification is very useful, right? So yeah, quick question at the back. Yeah. I'm kind of curious about scale. So what if the spreadsheet is like millions of rows, hundreds of thousands of columns, right? Or just like any sort of database. In that type of situation, how would you go in a cloud searching if there's obviously a context limit you have to copy for? Yeah, this is great. I probably should have done the spreadsheet example as my coding example. For a preview, my coding agent is a Pokemon agent. Probably a spreadsheet would have been better. Okay, the question was, what if the spreadsheet is very big? If you have a million rows, how do you think about? A hundred columns. Yeah, a hundred thousand columns or a hundred columns or whatever. Like, how do you think about it, right? Like, your database is also very big. Like, how do you do that? I think for all of these things, one, of course, as the data becomes larger and larger, it's just a harder problem. Like, you know, it just absolutely is. Your accuracy will go down, right? Like, cloud code is worse in larger code bases than it is in smaller code bases, right? As the models get better, they will get better at all of that. For all of these, I would think about, like, how would I do this? If I had a spreadsheet that was, like, a million columns and a million rows... what would I do? I mean, I would need to start searching for it, right? I would need to be like, like if I'm searching for revenue, I'd be like searching control F revenue, and then I'd go check each of these like results and I'd be like, is this right? And then like, I'd see like, is there a number here? And then I'd probably keep a scratch pad, like a new sheet where I'm like, hey, like equals revenue equals this, you know, and store this reference and keep going. So I think that's a good way of thinking about it is like, The model should never read the entire spreadsheet into context because it would take too much. You want to give it the starting amount of context. That's also how you work. Let's say that you open up the spreadsheet. What you see is rows is this. You see the first 10 rows and the first 30 columns or something. That's what you see. You don't load all of it into context right away. You probably have an intuition for, hey, I should load more of this. context right and and like oh I should navigate to this other sheet right and this other sheet has more data right but you need to like sort of you gather context yourself right and so the agent can operate in the same way it can like navigate to these sheets read them like try and like keep a scratch pad keep some notes and keep going so that's how I think about it yeah the back Yeah, so my question is about managing context pollution. It actually, I guess, relates to the previous question. Do you have a rule of thumb for what fraction of the context window you use before you start getting diminishing returns or if it becomes less effective? Yeah, the question is, context management, do you have a rule of thumb for how much of the context window to use before it becomes less effective? This is actually, I'd say, a pretty interesting problem right now. I think a lot of times when I talk to people who are using Cloud Code, they're like, I'm on my fifth compact. I'm like, what? I almost have never done a compact before. You know what I mean? I have to test the UX myself by forcing myself to get compacted. Just because I tend to clear the context window very often when I'm using Cloud Code myself. Just because, at least in code, the state is in. the the files of the code base right so let's say that I've made some changes uh Cloud Code can just look at my git diff and be like oh hey these are the changes you've made it doesn't need to know like my entire chat history with it you know in order to continue a new task right and so in Cloud Code I clear the context very very often and I'm like hey look at my outstanding git changes I'm working on this can you help me extend it in this way right that's like a way of thinking about it and um When you're building your own agent, like let's say we're building a spreadsheet agent, it gets a little bit more complex because your users are less technical, right? And they don't know what a context window is, right? That is like, I'd say, a hard problem. I think there's like some UX design there of like, can you reset the conversation stage, right? Like, can you, maybe every time the user asks a new question, can you do your own compact or something? And can you like summarize the context? Does it, like in a spreadsheet, A lot of the state is in the spreadsheet itself, so it probably doesn't need to know the entire context. Can you store user preferences as it goes so that you remember some of this stuff? There's a lot of, again, it's an art. There's so many different angles and ways in which you can do this, right? But yeah, you are trying to minimize context usage. You probably don't need sign-a-million context or something. You just need... good context management like UX design. Yeah. I just wanted to ask, the sub-agents were made to protect the conduct of the core agent, right? That's right, yeah. Sub-agents were made to protect the core agents. Would the spreadsheet be able to use multiple sub-agents and try to make a process so we chunk up the spreadsheet in the case where it's super large so then the agents can kind of run through each portion in parallel with each other? Yeah, yeah. I mean, yeah. So like, one of the things I love about Cloud Code is that we are like, the best experience for using sub-agents, especially sub-agents with bash. It is very, very good. I didn't quite realize all the pain. I think if anyone's going to QCon, I believe Adam Wolfe is giving a talk on QCon about how we did the bash tool. Adam's a legend and the bash tool did such a good job. When you're running parallel sub-agents at the same time, bash becomes very complex and there are lots of like... like race conditions and stuff like that. And so there's a lot of work that we've solved there, right? So this is like one of the things I love about Cloud Code is you can just be like, hey, like spin up three sub-agents to do this task and we'll do that. And in the agent SDK as well, you can just ask it to do that. So number one, sub-agents are great primitive in the agent SDK and I haven't seen anyone do it as well. So that's like a big reason to use it. yes generally you want it you want these sub agents to preserve context let's say you have if you have a spreadsheet you could potentially have multiple read sub agents going on at the same time right so maybe the main agent is like hey can this agent read and summarize sheet one can this agent read and summarize sheet two can this region summarize sheet three and then they return their results and then the agent maybe spins off more sub agents again right so this is like another knob you have um and i think what i want to say is like We've talked so much about all these different creative ways that you can do things. This is the level at which you should think about your problem. You should not really, in my opinion, think about how do I spin off a process to make a subagent or the system engineering between what is a compact or something. We take care of all of this for you in the harness so that you can think about what sub-agents do I need to spin off, right? And like how do I create an agentic search interface and how do I like verify its work? These are the really core and hard problems that you have to solve and any time you spend not solving these problems and solving like lower level problems you're probably not delivering value to your users, you know? And so yeah, I think sub-agents, big fan of the agent SDK sub-agents. Yeah, yeah, quick question? So we have this action and the verification task. So where exactly we need to put the verification? In this example, let's say after generation of the SQL query, I can verify if the right query is generated or not. That is the one path. Second path is like generation the query, directly executing. And once I will get the output, then I will do the verification. And how agent can choose dynamically which one is the right path? Yeah, so the question is like where do you do verification? Is it only at the end? Do you do it in the middle? Like things like that. I would say like everywhere you can. Just like constantly verification, right? Like I said, we do some verification in the read step of Cloud Code, right? So that's like a great example. You can do it at the end. You should absolutely do it at the end. But at any other point, if you have rules or heuristics especially, like if, for example, you're like, hey, one of my rules is that you shouldn't do like the total number of columns you should search should be under 10,000 or under 1,000 or something. That's like a nice way of doing it, right? Like similarly here, like maybe you should have been inserting like a huge like row of values, like give feedback to the model, be like, hey, chunk this up, right? You throw an error and give it feedback. And the great thing about the model is like it listens to feedback. It will read the error outputs, right? And then it'll just keep going. So yeah, verification is definitely like, I know I have it in this like as a, sort of a loop but um it's definitely more like verification can happen anywhere and and should happen anywhere like like put it in as many places as you can so um all right i do need to start doing some of the prototyping but i'll take one more question so right here yeah how do we say how do we form the steps how do we see the agent go search first yeah how does a loop actually start from the end You just tell it. So like, uh, yeah, the system prompt. Yeah. So like with cloud code, we just give it the bash tool and we're like, Hey, like gather context, read your files, uh, do stuff like run your LinkedIn, you know what I mean? Um, and so, yeah, again, with the agent, you don't need to enforce this, right? You don't need to tell it. Hey, like you need to do this because like sometimes it might not be necessary, right? Like let's say that someone is asking a read only question. for your spreadsheet, you don't need to verify that there are no compile errors, because you haven't done any write errors, write operations. So let the agent be intelligent, and in the same way that you would like that same freedom when you're doing your work, you're trapped in this box or whatever, same way. So OK, cool. I do want to try and see if I can do some prototyping now that we have this. the holder as well. Okay, yeah, execute, lint, we've done a bunch of Q&A. Okay, prototyping, okay, let's say that you have an agent, right? Like you want to build an agent, you come out of this talk, and you're like, great, I have a bunch of ideas, how do I do this? I think what I'd say overall is like, building an agent should be simple, your agent at the end should be simple, but simple is not the same as easy. right so like it should be very simple to get started and it is just go to clog code give clog code some scripts and libraries and uh custom cloud.md then ask it to do it right that's what we're going to do right um that's like it should be so easy to be like hey this is my api this would be like an api key uh can you like go search like you know I don't know, like my customer support tickets or something and organize them by priority or something like that, right? And then look at what Cloud Code does and iterate on it, right? And this is like a great way of like just skipping to like the hard domain-specific problems that you have, right? So you have a lot of like domain problems, like how do you organize your data, your agentic search, how do you like create guardrails on your database? These are all questions that you can just start solving right away with Cloud Code, right? And so... try and build something that feels pretty good with Cloud Code. And I think generally what I've seen is that you can do this and get really good results just out of the bat using Cloud Code locally. And you should have high conviction by the end of it. And so, yeah, I think like, I forgot more info, watch my AI engineer talk. This is like a deck for internal that we're using. Okay, so. Yeah, I'm gonna be inserting this. So yeah, you're getting what we show customers, right? So, okay, yeah, so yeah, use cloud code. Again, simple, but simple is not easy, right? So like the amount of code in your agent should not be like super large, doesn't need to be huge, doesn't need to be extremely complex. But it does need to be elegant. It needs to be what the model wants. You want to have this interesting insight. Let's turn the model into a SQL query. Or let's turn the spreadsheet into a SQL query and then go from there. So think about it that way. And Cloud Code is a great way of doing that. So, okay. Let's make a Pokemon agent. This is what we're going to do. Pokemon is a game with a lot of information. There are thousands of Pokemon, each with a ton of moves. We want to be pretty general, and so there is actually a Poke API. And the reason I chose Pokemon is just because I know that you guys have your own APIs as well, and they're all very unique. So I wanted to choose something with a kind of complex API that I haven't tried before. So the Poke API has, you can search up Pokemon like Ditto, you can search up items and things like that. And so it's got this like... Yeah, this custom API, you've got everything in the games, right? So, and yeah, like one of the quest things your agent might want, your user might want to do is make a Pokemon team, right? I love Pokemon. I know very little about making an interesting Pokemon team for competitive play. Could my agent help me with that? That'd be cool, right? So my goal is to make an agent that can chat about Pokemon and then we will like see what we can do and how far we get. So I've done some of this work already and I will open up and show you. So the first step and the prompt here is like, the first step is I'm gonna do mostly code generation for this, right? And so, let me. Is that gonna be on GitHub somewhere? It actually is. Yeah, it's on my personal GitHub. Oh yeah, I was going to commit all of this as well. Yeah. Yeah, yeah, so I think my personal GitHub is, let's see. All right. Is it secure GitHub or does it have malware in it? You guys are AI engineers. Yeah, like if you can get Owen, that's your fault. Yeah, so yeah, you can clone this dude like. I need to push the last changes. So, okay, so, yeah, can you guys see this? Should I put it in dark mode instead, or is this fine? Dark mode. Dark mode, okay. Okay, is this better? No? Do you want a different dark mode? Okay, I don't think this is good. It's good, guys. Okay, let's see. How does this work? Can you guys feel like you're in it? Okay, so here's an example of like, I've taken, the prompt I gave it was, hey, go search PokeAPI for its API and create a TypeScript library, right? And so this is all Vibe Coded. And so you can see here that it's created this like interface for Pokemon, right? And so it's created like this Pokemon API. I can get by name. I can list Pokemon. I can get... all Pokemon I can get species and abilities and stuff like that and so like this is just a prompt that I gave it right and generated this like TypeScript API it also did it for moves and then it's created this like it's created this like API that I can use import PokeAPI right from the PokeAPI SDK and yeah you can see like sort of how it's like set this up and Now in contrast, right, and so this is the cloud.nb, right? This is a TypeScript SDK for the Poki API. This is like the modules in the Poki API. Here are some of the key features. I'm asking it to write scripts in the examples directory, and then it will execute those scripts to help me with my queries, right? And I give it some example scripts. It doesn't always need all this information, right? But yeah, fetching Pokemon, listing the resources, getting data, things like that. So this is like my agent, really. It's like a prompt I gave it to generate a TypeScript library and then this cloud.md, and I can chat with it in cloud code. I'll also show you a version of it that is just tools, right? So here I'm using the messages completion API, right? And I've given it a bunch of tools from the API. So like get Pokemon, get Pokemon species. Get Pokemon ability, get Pokemon type, get move. So you've defined all of these tools and you can see that like, you know, I also just gave it a prompt and told it to make the tools. It doesn't want to make a hundred tools, right? Like there's a ton of smoke on, or sorry, Poke API data. But like, you know, there's only so many parameters it can do. So it's got this like tool call and now And I made like a little chat interface with it, right? So let me now go here and say like, this is my tool calling. Great, so yeah, here we've got this chat.ts, right? I use bun when I'm prototyping stuff just because like, I don't want to compile from TypeScript to JavaScript. Again, Bun has linting built into it. It's a way of simplifying for the agent so the agent doesn't need to remember to compile. But TypeScript is better for generation because that's types. So I'm going to start this Bun chat and then I'm going to try, okay, what are the generation two water Pokemon? And you'll see that it's starting to search. And I'm logging all the tool calls here. This is very, very important because it needs to... do the tool calls and so you can see that what it's doing is like it's searching a bunch of pokemon um and then it told me okay here are the water pokemon for gen 2 right it's got totodile crockinaw for alligator you can see sort of like how it's thought like in between each step it's thinking through um the previous steps right now like let's say that i want to do with claw code i think i might need to uh I need to delete this example. Oh, yeah. SPEAKER 1, how do you log the tool calls? Is this just an argument? SPEAKER 1, oh yeah. This is in the normal API. So I just like in the model, every time it logs in, I just call this. This is in the normal Anthropic API. In the SDK, I'll get back to the SDK. It's just like you just log every system message. So it's doing console logs. Does that make sense? So the chat and I think you're showing, is that just using the regular API? Yeah, that's using the regular API. So not the agent SDK. Not the agent SDK. Yeah, yeah. And so what I'm going to do here is here, I'm going to delete the script because I don't want it to cheat. But OK, so here you know that I'm just opening clock code. I've created a bunch of files here. I'm going to say like, can you tell me all the generation 2 water Pokemon? And then we'll see what it can do, right? So I forget if I need to prompt a Tori script or something. I think I'll be fine. We'll see what happens. Do you mind going to the core SDK file and just showing, talk about getting context and then action and then verification? Can you show that in the code? and how we're configuring the tool description? Yeah, so we haven't done the SDK part yet. So, so far, I've just put some APIs in Cloud Code. Yeah, yeah, yeah. I'm sorry, I thought I missed that. No, no, no, yeah, yeah, yeah, of course. But yeah, so, okay, you can see here, it's given me a lot more, right? And, yeah, it's given me a lot more. So it's saying there's 21 people from mine. I think this is roughly right. I think it just knows. Anyways, Pokemon is slightly in distribution, which is good. What it will do is it will try and write a script and Because you don't want it to think as much, right? So here it's like, okay, what I'm going to do is, let's see, gen two, Pokemon, yeah. Okay, so yeah, you can see here, it knows like, okay, the start of the generations, it fetches these per API. I guess it's decided not to use like my pre-built API here. And then, yeah, and then runs it, right? I think I need to improve the Cloud.nb for this. But anyways, you can see that it's able to check 200 plus Pokemon and then check for their type and get their information. So this is just a quick example on how to do code gen and how to use Cloud code to do it. So we'll run this script and then keep going. it will give me the output. Basically what I want to show, let's see we have roughly 15 minutes left. Actually this is one of the demos I was thinking of doing. Cloud Code plays Pokemon. So let's say you want to do like an agentic version of Cloud plays Pokemon, how will you do it? What you would do I think is like it would give it access to the internal memory of the ROM. And so let's say that it wanted to find its party, it could search that in memory. And Pokemon Red is a very well in distribution, reverse engineered game. And so it could search in memory to be like, hey, these are the Pokemon. This is how I figure out where the map is. This is how I navigate it. So this is like maybe, actually, to the reader, if you want to try it out, it's like. There is like a Node.js GBA emulator. I think I have to legally say you have to go buy Pokemon Red and try it. But yeah, I think like, yeah, good example. Anyways, here. So it's fetched all of them and it's listed all their types. And yeah, you can see how it's like used code generation to do this, right? So a quick example of using Cloud Code to prototype this. Now there can be like more interesting like data here so um i do want to leave time for example so i think i'll just sort of like for questions so i'll just sort of go through like an example let's say you're making competitive pokemon competitive pokemon has a lot of different variables and data so this is like a text file from this online like a library basically which stores like all of the pokemon and their like moves and who they work well with and don't work well with and you know like who they're countered by and all of these things right so there's a ton of data here right and it's all in text cloud um which is actually pretty good for cloud code right because i can say like okay um hey i'm gonna give it a little bit more data normally put this in the um check the data folder tell me i i want to make a team around venusor Can you give me some suggestions based on the Smoldon data? Smoldon is like this online API. And so I'm not entirely sure what we'll do here yet. I haven't done this query before, but we'll see. I think it'll be fun. Yeah, but what I wanted to do is sort of graph through this data, right? And sort of figure out from itself, from first principles, not having seen this data before, how can I answer my query? While it does that, I'll take any questions. Yeah? So this is like really on top of Cloud Code. And so my question is, if we were to deploy this custom-facing, are we supposed to have Cloud Code running in like a swarm? Or are we somehow able to put Cloud Code part out? just use bot and the agent SDK. Yeah, so let me show you very quickly what it looks like to use the agent SDK here. So I've already done this file system. And again, I want you to think about the file system as a way of doing context engineering. This is a lot of the inputs into the agent. So my actual agent file is like 50 lines. And it's mostly just like random like for the plate right like I guess yes decided to stop it from Writing scripts outside of that custom script directory again fully back-coded so yeah, you can see like it just runs this query takes in the working directory and Like like runs it in a loop, right and so Probably I'd want to like turn into like some allowed tools here and stuff, but it's very simple and so If I were to like productionize this, the first step I do is like, okay, I've tested it on cloud code, it seems to do pretty well. I write this file, then I put it, there are two ways to do it. So one is, I do think that like local apps might be coming back with AI because I think that like there's such an overhead to running it. Like for example, cloud code is a front end app, right? Like it works on your computer. So maybe the way I ship this as a Pokemon app is like, hey, I have an app that you install and it works locally on your computer and it's writing scripts. I think that's one way of doing it, right? The other way is, yeah, you host it in a sandbox. And again, there is a bunch of different sandbox providers that make it really easy. Cloudflare has a good example of using the agent SDK. And it's just like sandbox.start, and then like bun. agent.ts and that's kind of all it takes, right? Like they've abstracted away a lot of it. So you run like the sandbox and then you communicate with it. And yeah, I think there is like some very interesting stuff that I'm not sure I had time to get to, but like I think some interesting questions are like Yeah, like how do you do this sort of like service now? We're spinning up a sub like a sandbox per user There's a lot of like I'd say best practices to solve here one thing I just want to call out for you guys to think about If you're making an agent with a UI like let's say that you have Yeah Pokemon agent and I wanted to have an UI that is adaptable to the user, right? Like maybe some users are doing team building, some users are helping with their game, some users just want pictures of Pokemon. How would I have an agent that adapts in real time to my user, right? The way I would do it is in my sandbox, I would have a dev server, right? And the dev server would expose a port. It would run on BUN or Node or something. like expose the port the agent could edit code and it would live refresh and and your user would be interacting with that website this is how a lot of like site builders like lovable and stuff work right they use sandboxes and they ascent host essentially a dev server and so thinking about this for your user if you want a customized interface this is a great way to do it um okay let's see let's see what it did um okay Cool, okay, so it's like written this script, it's generated, like, showed me some base stats and suggested a, like, a moveset and some teammates and you can see sort of like, let's see, what did it do? Control E. Yeah, okay, so you can see here what it started doing is like it started searching for Venusaur, right, and it started finding those types, like those Pokemon, and when it does that it also gets other Pokemon that mention Venusaur, so it gets like its teammates and its counters and stuff, right? And it's sort of over this time found interesting Pokemon, right, that like it might work with, right? So it's done a bunch of these searches and it's got this profile, it's found its most common teammates and written a script to analyze it, right? And so this is all based on a text file. Of course I could have pre-processed a text file. a little bit more. But yeah, it's like done this sort of like interesting analysis for me. And again, I'll push up more code to the GitHub repo and also tweet about this. I'm on Twitter. I'm trq212. I tweet a lot. So definitely like mostly by agent SDK stuff. But yeah, we have about eight minutes left. So I want to spend the rest of time taking questions about kind of anything, you know, and I'm sorry we didn't get to do. more prototyping. But yeah. I was going to say, with the Cloud play, can you plug this in with that just to see if the agent will be more selective with the teaming to try as a character? Yeah, we'll put it in Cloud play as Pokemon. Yeah. I do want to make Cloud code play as Pokemon. I think that would be fun. Yeah. I think Cloud play as Pokemon. I think we try and keep it like a pure reasoning task as much as possible. Other questions? Yeah. Yeah, I do think overall, especially right now, agents are kind of pricey, you know what I mean? Because like the models have just started to get agentic. We really focus on like having the most intelligent models, you know, and like you generally, this is just like an overall like fast. business software thing you'd rather charge fewer people more money that really have like a hard problem you know and so i think this is still good like you probably should find um you know these hard use cases but i would say like number one make sure you're solving a problem that people want to pay for right it's like the number one step right and then number two um yeah i think you could do subscription or token based i i think this kind of comes down to like how much you expect people to use your product uh versus like how much You expect them to use it occasionally. Cloud code, obviously, people use a lot. And in order to, we do a mix of, if we give you some rate limits, and if you exceed it, we do usage-based pricing. I think that it's very dependent on your own user base and what they will do. But I will say monetization is something you should think about up front and design your agent around, because it's hard to walk back these promises. Yeah, back there. I haven't heard you talk at all about hooks. Yeah, there's so much to talk about. Hooks are great. We do ship with hooks. Hooks are a way of doing deterministic verification in particular or inserting context. So, you know, we fire these hooks as events and you can register them in the agent SDK. There's like a guide on how to do that. Examples of things you might use hooks for is like, for example, yeah you can run it to verify the like a spreadsheet each time uh you can also let's say i'm working with an agent and i'm the agent is doing some spreadsheet operations and the user has also changed the spreadsheet this is an interesting like place to use a hook because you can be like hey has after every tool call insert changes that the user has made uh and so you're giving it kind of live context changes um in an interesting way so um Yeah, I think, yeah, there's more stuff on like the docs about hooks. I am happy to like talk about it afterwards as well. Yeah, more questions? Yeah. Yeah. Yeah. Yeah. Okay. Yeah, sure. Yeah, so like let's say you've done this prototyping, you found something that works. What I would do is like somewhere the cloud.md, like obviously like when I tried doing this one time it like didn't use my API directly and it wrote JavaScript. I should have been more specific in my cloud.md to be like, hey, you should use this. Yeah, I think like so that's one thing. The second thing is, yeah, do some reason to have the helper scripts that you need and then write something like this agent.ts script to run the . Yeah, more questions? Yeah, in the brain? Yeah, this is a good question. I think there is some messiness, right? I think one of the things, if an agent knows an answer... and you want to fight it, kind of, to be like, okay, it's generation nine now, and Venusaur's stats have changed, and there's this new... This is hard, I actually think. One of the ways of doing that is hooks. So you can say, for example, like, hey, don't... If you've, like... return a response without writing a script, you can check that, you can be like, give feedback to be like, please make sure you write a script, please make sure you've read this data, right? And you can use hooks to give that feedback in the same way that in Cloud Code, we have these rules, like make sure you read a file before you write to it, right? So add some determinism. It can definitely be, like I said, it's an art, sometimes, yeah, maybe like riding a horse, I guess, probably. Yeah, and the gray. How are you guys dealing with large code bases? I'm working like a 50 million plus nine code base and so the correct tool doesn't work really. So I'm having to build my own semantic indexing type thing to kind of help with that, right? Sure. Is there any kind of add-in product maybe, thinking about how that can be more native to the product? In a couple of months, is the thing I'm writing just going to go away? Or how do you guys think about that? Okay, your last question in a couple months as you're thinking to go away. Generally, yes. Yeah, any time you ask about AI, yeah. I think that semantic search, this is a cloud code question, more than a genistic question, but happy to answer it. We, you know, they're trade offs of semantic search, it's more brittle. I think you have to like index and search and it's not necessarily, the model is not trained on semantic search. And so I think that's sort of like a problem, like, you know, graph, it's trained on because it's like, it's easy to do that. But like semantic search, you're implementing your bespoke query. For like very large code bases, you know, we have lots of customers that work in large code bases. I think what I've seen is sort of like, they just do like, good clon.mds, you start in, you know, try to make sure you start in the directory you want, have like good like verification steps and hooks and links and things like that and so, you know, that's what we do. We don't have, you know, a custom, we dog food clon code, right? So, yeah. Okay, yeah, last question. We have to close unfortunately actually. Let's give it up for Tariq everyone.

Learn to use Anthropic's Claude Agent SDK (formerly Claude Code SDK) for AI-powered development workflows!

https://platform.claude.com/docs/en/agent-sdk/overview
https://x.com/trq212

**AI Summary**
This workshop by Thariq Shihipar (Anthropic) details the architecture and implementation of the **Claude Agent SDK**. The session moves from high-level theory—defining "agents" as autonomous systems that manage their own context and trajectory—to a live-coding demonstration. Shihipar builds an agent "Harness" from scratch, implementing the core **Agent Loop** (Context  Thought  Action  Observation), integrating the **Bash tool** for general computer use, and demonstrating **Context Engineering** via the file system to maintain state across long tasks.

**Timestamps**

00:00 Introduction: Agenda and the "Agent" definition
05:15 The "Harness" concept: Tools, Prompts, and Skills
10:10 Live Coding Setup: Initializing the Agent class and environment
15:45 implementing the "Think" step: Getting the model to reason before acting
25:20 The Agent Loop: connecting `act`, `observe`, and `loop`
33:10 Tool Execution: Handling XML parsing and tool inputs
42:00 The "Bash" Tool: Giving the agent command line access
49:30 Safety & Permissions: "ReadOnly" vs "ReadWrite" file access
58:15 Context Engineering: Using `ls` and `cat` to build dynamic context
01:05:00 The "Monitor": Viewing the agent's thought process in real-time
01:12:45 Handling "Stuck" States: Feedback loops and error correction
01:21:20 Multi-turn Complex Tasks: Building a "Research Agent" demo
01:35:10 Refactoring patterns: "Hooks" and deterministic overrides
01:48:39 Q&A: Reproducibility, helper scripts, and non-determinism
01:50:31 Q&A: Strategies for massive codebases (50M+ lines)
01:52:00 Closing remarks and future SDK roadmap

* **Evolution of AI Capabilities:** Shihipar argues we are shifting from **LLM Features** (categorization, single turn) to **Workflows** (structured, multi-step chains like RAG) to **Agents**. He defines agents as systems that *"build their own context, decide their own trajectories, and work very autonomously"* rather than following a rigid pipeline.
* **The Claude Agent SDK Architecture:** The SDK is built directly on top of **Claude Code** because Anthropic found they were *"rebuilding the same parts over and over again"* for internal tools.
* **The Harness:** A robust agent requires more than just a model; it needs a "Harness" containing Tools, Prompts, a **File System**, Skills, Sub-agents, and Memory.
* **Opinionated Design:** The SDK bakes in lessons from deploying Claude Code, specifically the "opinion" that general computer use (Bash) is often superior to bespoke tools.


* **The Power of the Bash Tool:** A key technical insight is that the **Bash tool** is often the most powerful tool for an agent. Instead of building custom tools for every action (e.g., a specific API wrapper for a file conversion), giving the agent access to the shell allows it to use existing software (like `ffmpeg`, `grep`, or `git`) to solve problems flexibly, similar to how a human developer works.
* **Context Engineering:** Shihipar introduces the concept of **Context Engineering** via the file system. Instead of just "Prompt Engineering," the agent uses the file system to manage its state and context.
* **Files as Memory:** The agent can write to files to "remember" things or create its own documentation (e.g., `CLAUDE.md`) to ground future actions.
* **Verification:** The file system serves as a ground truth for the agent to verify its work (e.g., checking if a file was actually created).


* **The Agent Loop & Intuition:** Building a successful agent loop is described as *"kind of an art or intuition"*. The loop generally follows a **Gather Context  Take Action  Verify Work** cycle. Shihipar emphasizes that this loop allows the agent to self-correct, a capability missing from rigid workflows.
* **Strategies for Determinism (Hooks):** During the Q&A, a technique for controlling agent behavior is discussed: **Hooks**.
* If an agent hallucinates or skips a step (e.g., guessing a Pokemon stat instead of checking a script), a hook can intercept the response and inject feedback: *"Please make sure you write a script, please make sure you read this data."*
* This enforces rules like "read before you write" without retraining the model.


* **Scaling to Large Codebases:** For massive codebases (50M+ lines), standard tools like `grep` or basic context window stuffing fail.
* **Semantic Search Limitations:** Shihipar notes that while semantic search is a common solution, it is *"brittle"* because the model isn't trained on the specific semantic index.
* **Solution:** He recommends good **"Claude MD"** files (context files) and starting the agent in a specific subdirectory to limit scope, rather than trying to index the entire 50M lines at once.