Simplifying Context Engineering for AI Agents in Production

Thank you everyone for joining us here to talk about context engineering. First, let's start off with a little bit of housekeeping. We're going to shoot for around 45 minutes. I will tell you that between Josh and I, we are never at a loss of words. So we will try to stick to 45 minutes. But if we go over a little, I hope you'll forgive us. um for q a please use the q a panel there should be a q a panel available to you um we inevitably get questions in the chat as well we'll try to address them there but to help josh and i and some of our colleagues out if you can put your questions in the q a panel that would be super super helpful and yes yes yes the first question is always will we get the recording of course you will get the recording um we will send the recording and and a number of other things out to the registrants after the webinar concludes. So with that, let me introduce myself. My name is Cornelia Davis. I'm a developer advocate here at Temporal. I have been here for a bit now. totally the newbie anymore. My personal background is I've spent the last decade or more, probably closer to two decades in kind of the distributed systems microservices space. Just this year have been a part of Temporal and super thrilled to be here. And I'm super delighted to be here today with my friend and colleague, Josh Smith. Josh, you want to introduce yourself? Yeah, I'm a solution architect here at Temporal. I've been here, I think my two-year anniversary is like today or yesterday or something. So that makes me almost an old timer here at Temporal. And my background is in building distributed systems. I was thinking about giving this intro and I was like, oh man, it's been almost 20 years that I've been working in this space. But certainly this year and a little bit. years before I've been working in AI and helping people, you know, work with models and build agents and build tools and helping our customers with that. So that's my intro. Super. Congratulations. Happy anniversary, whether it's today or tomorrow. That is a big milestone. Yep. Super cool. all right so what we're going to talk about today is we're going to spend just a little bit of time kind of talking about context engineering in general and then we're going to go through a number of different scenarios things that you need to be wary of i won't belabor i won't read the words here because we're going to go through sections one two and three in some level of detail with lots of demos so i promise you we aren't just going to slide you to death Then we'll do a little bit of a wrap up and give you some resources where you can find some more information and invite you to our next event. And you'll see what our next webinar, AI based webinar topic is going to be. So stay tuned and we will show you a number of other things. By the way, some of you probably have noticed already we have. posed a number of poll questions these are really really helpful for us to get a sense of who our audience is and also there'll be a few more poll questions toward the end it's an opportunity for you to give us feedback on what you'd like to see more information on as well so please please please if you have a moment while you're listening in on this do respond to those polls they're super helpful Alright, so let's jump into some context engineering basics. And so for those of you who... have not heard the term although i suspect most of you have at least heard the term there's this notion of context engineering and you can think of it as a superset of prompt engineering of course we've been talking about prompt engineering for gosh since the very beginning and if you remember there was single shot there was multi-shot where we started realizing that we maybe wanted to follow up And it was all about, if you're just asking something, if you use five words to ask the LLM about something, you're going to get one response. But if you use maybe two or three sentences, you tend to get a little bit better information. Well, we take that prompt engineering and we extend it as we get into more sophisticated interactions with the LLMs. And that extension is not just about the user prompting and the use the things that the human is saying but it's also about bringing in a bunch of other contexts that is going to help the llm make a reasonable decision now we're going to talk about what you bring in and the way that you bring it in but having that structured having all of that context that you're bringing in and and we also of course context has the the similar thing with the context window. What we're really talking about here is what do we put in the context window so that the LLM can do the best job that it can. But it's not only the gathering of all that information into a single shot or carrying that context through a bunch of different interactions with the LLM. It's about managing that context. It's about what do you bring in? What do you throw out? so that the LLMs can be making good decisions. So, you know, in summary, prompt engineering was just about a single prompt. Context engineering is about ongoing prompt engineering plus some of the other context. So thanks for that click. Yeah. So why does it matter? Well, I think I've already spoken to a lot of that. It's we want to get good responses out of the LLM. and um and we can only do that it's only as good as the inputs that we put in there um i put a little you know little heart thing there because this was my dear friend josh who In that final phrase, I won't read the whole slide to you here, but it says, you know, in some ways, a good agent context engineering creating good agents production is good human context engineering. So I love that corollary, Josh, to, you know what, if we give humans way too much, then they get confused and they can't respond in a particularly good way. Yeah. I always like the human computer analogies, you know. And if you're sending too much information to a human and telling them to do 500 things, they can get a little overloaded. And I felt a little way, we'll get more when we get into the demos about this with the agents that I had written. And then I did bad context engineering to kind of break them on purpose. And I felt bad for them. Like, oh, no, I would never ask myself to do this. Why am I doing this to you poor agents? But they did okay. And we'll talk about that when we do the demos. you know the whole goal here for me is getting working useful agents in production to help people solve the people problems and and to do that you got to have good context engineering and build good agents that give people the right context they need to do their jobs as well yeah i love how you personified the uh llm the ai oh yeah oh i'm doing these horrible things to you oh yeah i if you While I was building this out, I was talking to my wife and I was like, I feel so bad for these poor little agents that I'm doing this to. I just felt guilt. So maybe when the AI overlords rule us all, they will have mercy on me. Yeah, there you go. I'll keep going. Very good. I love this picture. And you can see this is not a picture that we've created. It comes out of Anthropic. And I think that this really kind of summarizes what we've been talking about here. Prompt engineering on the left-hand side, pretty straightforward stuff. context engineering on the right hand side much more involved you can see that the context that we're sending into the llm here is made up of a bunch of things it's not just user messages and assistant messages but it is documents it is mcp tools which i think josh is going to show you some of those things it's instructions it might be some rag information all of those types of things and then importantly you notice that the curation arrow there that's what i meant by ongoing maintenance so we're constantly refining that context as we go through the loop now we're going to show you some techniques through the rest of this that are going to make the mechanics of some of that easier for you But I'll be frank with you, this context engineering problem of deciding exactly what the right context is that you should be putting in and taking out, it is, if you will, the new programming. Because it's really, truly like programming and you're essentially programming the context. And that's one of the ways that I like to think about it. So I love this image. Yeah, me too. I think one thing that's driven the rise of context engineering just as i'm looking at this slide anew today is the idea that we can use tools and information like you were saying cornelia on the fly to get more information for our agents as they progress but that opens up a whole new set of how do we do that uh problems and i think that's really what we're going to focus on today yeah perfect And then another fairly well-known slide, I've seen it on all sorts of social in lots of different places, but there's all sorts of problems that can happen. So if you don't do your context engineering in a particularly effective way, you can end up. poisoning your context you can distract the llm with all sorts of stuff you can have context clashes so first i wanted to go to bermuda and now i've changed my mind and now i want to go to hawaii or something like that so we we have to keep managing those things and of course we can get confusion now there's a lot of ways that you can fix it um and sure there's things like you know having some content, you know, ragging your content, loading tools, unloading tools, pruning the context. And you'll see both Josh and I are leveraging some of these techniques in the demos and the use cases that we're showing you here. But what we want to do is not talk about those things in the abstract, but really talk about those things in context. So in this, by the way, this list is not exhaustive either. That's another thing to keep in mind. Yeah. I didn't know if you wanted to jump ahead or stick more on that side, but I jumped ahead. Yeah. No, that sounds great. So these are the ones that, these are the best practices that we're going to talk about. We're going to talk to you about scoping your agents. So this is what Josh was saying earlier is like, if you give the agents too much to do, you know, five things rather than one thing, you know, have it do one thing and do one thing well that can be problematic so josh is going to really drill in on that scoping your agents properly and then even once you've got your agents scoped properly you still have to address the problem of making sure that they get the right data at the right time and then finally we're also going to talk about that managing the history so whether it's a chat interface back and forth with the user or it is the agent going through multiple loops and continually adding to the historical context it's managing that growing history so those are the three use cases that we're going to talk about so josh is going to handle the first couple of them and then i'll jump in later on so josh take it away okay sounds good um so the first thing we're going to do is be really mean to our agents um and we're gonna We're going to make our agents basically do too much, think too much. And I have an existing system. We'll talk through it in a little bit of detail. But basically, the takeaway that we're going to do here is we are going to, instead of having agents dedicated to specific tasks, we're going to throw everything at a single agent and see how it does. And LLMs get better all the time. What I've found is this agent basically became unpredictable so sometimes it works fine and sometimes it gets real confused and we'll show that in the demo i think um i'm getting a little bit to to my takeaways but what we found at temporal is agents work best when you give them a very specific concise, focused amount of data and prompt. If you can narrow that context in your context engineering and give the agents focus, they tend to perform more reliably. What we're going to do is we're going to take a working system and break it. What I have is a system that detects problems in my order management system, such as not enough inventory. does this order need manager or you know special approval or is there late payment we got grumpy customers what are we going to do about it and right now i'm going to show it to you working and then we're going to break it it has a detection agent that agent has a tool that loads order data and their statuses pretty simple then we've got an agent that loads the the detected problems the order data and inventory data and says hey let's let's dive a little deeper let's look for specific problems um and then and then with the problems we have a planning agent that knows about the problems and tools and basically marries problems and tools and says okay for this problem let's run this tool for this problem let's run this tool so on and so forth um then we have an approval step that's going to be me that will lead to execution of tools and then there'll be a report agent and then what we're going to do is we're going to throw out that beautiful separated little agent architecture or micro agent architecture and do what we call a monolith agent which does tries to do everything all right so I'm going to switch over to my my IDE here and what I've got is I could go into the code. We'll share a link to it if you want to go into the detail. But what I'm really going to do is start my monolith agent and it's going to run. Oh no, don't do that. I hit the wrong thing. I'm going to start my regular agent and it's going to run and hopefully I didn't get too far ahead. I've got my workflows here. This is temporal. This is a temporal webinar. I can see in temporal that it's running and it did my detect, analyze, planning, and now it has a timer and it's waiting for approval. I'll go back to my CLI and show what that's looking like. Usually I do this demo with a UI and MCP and stuff, but I'm really keeping it simple for this. What's interesting, I don't know if you caught it, but my plan repair, even in my dedicated agent um or my micro agent did fail once and that's kind of a fascinating piece of temporal is it'll automatically retry so that happened i can get my last failure but the point is it's waiting for me to approve and if i go back here um i can see in my command line i've got this is in the harry potter universe i thought that was funny harry potter's order um he needs maybe a payment reminder um hermione needs to uh order some spew badges if you're a harry potter fan you might know why she might want that and we don't have enough on our inventory so the agent has told me i need to run these tools um and then just one little nice thing about temporal is there are queries that we can add to our workflows so for an agent um you can add all kinds of really nice little queries that'll tell you about that agent such as um let's get how confident it is that there are problems i added that query or um what are some other good ones let's get some analysis you know we can see the analysis that the analysis agent outputted and get a little more detail if we want to do debugging at any time in the workflow but i have run this agent a lot i know the problems these are right um you know Hagrid does need approval for his restricted items I've reviewed them and I'll say yes with my little command line agent and it says repair approved by user and I can go back to my workflow and we did a little human in the loop we sent a signal to approve that repair and then it executed the repair agent or executed the repair tools and then did the report agent and I can see that that worked and I can get my results This says there's a repair report. That's something that I had my repair agent do is basically an agent take all of the information from the tools and the current state after the tools updated my inventory database and my order database and do some analysis. I loaded in that data with tools and then I but read tools. And then I created this report and it, and so I had a planning report and then I have the repair report. So I didn't review the planning report before, but here's my repair report. And I can see, oh, you know, we did some stuff and it worked and we're in business. So, okay. So that was our, our first, our first everything's. yeah um quick question so there's a great question in the chat if you don't mind um the question uh is how did you modularize agents in a single temporal workflow how did they host them i apologize if i'm asking an obvious oh yeah so it's about modularization yeah that's a great question so temporal gives you a lot of i'll say orchestrative power um and With temporal, what I did was, and I think there's a link to the architecture that's in the chat, but I basically made all of these agents their own temporal activity calls. There's a workflow that orchestrates how they work together and pass data back and forth. Then the detection agent is an activity call that loads the data that it needs, then calls the LLM, and then processes that data. I can do data validation and fail. if my data isn't valid. That prevents me from poisoning downstream systems. Every one of these agents essentially is their own activity call except for the plan execute phase because I want to plan and then hold and wait for human input and do human in the loop. Temporal workflows make that super easy. One of my agents conceptually is the whole workflow that orchestrates these subtasks, waits for human input, and then when they get approval, executes the writing tools and then executes the report agent. I hope that's a good answer to your question, Ayusha and the people in chat. Yeah, I think that was really good. I'm going to call out a phrase that I picked up from you, Josh, which is, Agents as, now my brain is freezing. So tools as agents, agents as tools is something that I've heard you say several times. And we're not, you didn't necessarily go all the way to NCP here, which I think is totally fine. I mean, sometimes you're going to have those. MCP tools are going to be themselves sub agents and or inactivity can act as a tool as well. And that's what you've done. And you've made those kind of sub agents. Is that right? Yeah, that's right. Cool. Yeah. If you look at the video attached to that blog, I actually use this whole thing as a long running MCP tool for a different agent entirely. So there's. I didn't want to get too complicated for today's demo, but if you want to go deep down the rabbit hole, there's more info there. There's another great question I saw that I think would be good to answer. Is that summarized as one micro agent per activity in the workflow? Yes. What's really, really great about that and why I wanted to call it out is because my agents effectively map to activities, I can get agent input and output. through the temporal workflow and temporal visibility. I can see the input to my agent and the results. Because temporal does this for every workflow, I can have all this wonderful visibility into my agents as they're running or as they've completed as this one has. That worked. We did results. It totally worked. What I'm going to do now is reset my inventory and orders data and do the same exact thing. I've done this test a bunch of times and it's always exciting to know what's going to happen when I mush everything together into my monolith agent. Let's find out what happens. One thing you might be thinking is, well, There's no human approval step in the middle. This is just one agent that just does everything. The way I wrote this agent was if its confidence score was high enough, it doesn't wait for human approval. It's just one one-shot agent basically. But if its confidence score is high enough, it will run the tools. It did do that. Let's go look at what that looks like in my workflow. I called it my monolith agent. It's just one activity. If you look at the inputs, sorry, that's my workflow. Let's go to the activity. Got my inputs and then I've got all of my outputs and it ran, but I'm sort of like, well, what did it do? I don't have any separation between all of the steps and I don't know what it did. and i don't have a report agent either and that's kind of a bummer but it did stuff i know it did and i had it like basically dump what it did as the last step um not using any agenda capabilities but just write out some markdown and turn it into a pdf of what it actually did so its confidence score was 0.82 that's a little lower but good enough and it was above my threshold so it did the repairs um and it kind of did the same thing um it seems like it was pretty confident that it needed to do some of these tools wasn't too sure about ron that's okay it knew it needed to prompt harry for his vault transfer from gringotts and it knew it needed to order spew badges and unfortunately it looks like it worked really really well When I was playing with this earlier, I did a bunch of tests of this and this agent kind of was all over the map, but it worked really well today, which is sort of a disappointing demo. That's kind of how it goes for me with temporal demos. When I want them to break, they work. So historically, you'll have to take my word for it. When I run this agent many, many times, I ran it with a bunch of different models. Generally, this is what happened when I ran it as a monolithic agent. the split out agents version many many times and it always works very predictably well but 60 of the time with the monolith agent approximately it found problems that didn't happen that didn't exist it was like hey there's a problem i'm a little confused about context and tools and everything you threw at me i'm going to make up problems because i'm lost it would also sometimes give confusing responses or be really really sure or really really unsure that it was running the right tool. None of that happened in the report today. I'm a little bummed. But that does happen. So yeah, I can't even make it break. And then deeper problems with writing your agents as one big monolith is, and I talked about this, I couldn't really tell what it was doing. When I say, take all the order problems you found, pick the tools for it, then go run those tools. there's some interesting thinking that I'd like to debug that I can't get at using the breakout agent model from my first architecture. It's all one shot, one LLM call. I basically have to trust that it did pick the right tools, did pick the right problems, looked at them the right way, and then all I get is the output of what it chose to do. That can make it harder to debug the agent, especially when it goes wrong. Then a couple other things that I noticed were if you have failures, it has to retry with the whole context window, all the data I loaded and mooshed together, and that can be potentially expensive. If any one part of the thinking fails, the whole thing fails. That's a bummer. The real big problem I found with this agent is it doesn't have that human in the loop step between planning and execution. That means that when i run my agent and say if you if you feel good about your tools just run them then it's sort of acting on its own um so that's kind of a bummer all right i'm gonna ignore the the chat because it looks like there's cool conversations here um so i'm gonna give a little bit of design guidance i make no claim to be the world's greatest agent builder i can only claim to have broken agents in lots of fun and interesting ways and this is some things that i learned from building agents big and small Like Cornelia and I have said, it's a really good idea to have your agents handled one single task well. If you're familiar with like monoliths versus microservices, maybe the term microagents will make sense to you. That seems to work really, really well. Then you can use workflows to orchestrate multiple agents and their context. We'll talk about context management a bit in the next section, but give the agents the tools and the information they need. They can load that in their activity calls. and and then pass that context around to help the agents do their work strong recommendation for me have a separate planning human in the loop and execution step and that way humans can approve and validate especially if you're doing right type tools tools that change data that can improve visibility and reliability i love creating reports that's super helpful to understand what was happening i was originally going to do this demo without a reporting step for the monolith agent, but I had to just write out some markdown just so we could see what it was doing. Otherwise, it was completely invisible what that agent was doing. It was really hard to tell. Create reports, use good tools to load context, and also to get that context out where humans can see it. Then finally, if you want to manage separate processes or different life cycles of agents, Child workflows or Nexus workflows are really nice to manage sub-agents and tool calls, especially if they're long-running. Right here, I had one long-running agent and then multiple short-running agents. So I used a parent workflow and then a bunch of activities. But if you have a long-running sub-agent, child workflows are great for that. All right. Cornelia, do I need to pause or do you want to jump in and give any clarification? No, I think it's good. We are actually starting to already run a little short on time. Let's just push forward. Okay, cool. The next problem I'm going to do is give all the context to my agents. This also makes me feel sad. Each agent in my original model loads its data. One thing I was going to do is demonstrate. um like not loading enough context but that was just so unreliable as to be not a useful demo just the agent fell down and died so i think i think everybody um that's watching this understands if you need your agents to take um take data and do something with it you've got to load that and give them access to that data use rag use some sort of of data orchestration to get data so we're going to do the opposite we're just going to keep throwing data at agents so the first agent is going to be the same it's going to load orders then the second order is an inventory then tools problems orders inventory this is how it works now but what we're going to do instead is have the same agent architecture but we're just going to keep adding more context and what you might think is since we're changing the orders and inventory data as we run our tools we're going to cause a conflict we're going to have our orders and inventory in our context window and we're not going to take it out we're going to leave it in there and model that we were using some sort of shared context architecture or just overloading context so we're going to introduce some context conflict con context poisoning so let's do that i'm going to reset my data files again. I tried to keep this demo as simple as possible, so you could just download it and run it. I don't have a separate database or anything. I just use JSON files. We're going to run our agent again, and you can see it's running. If I go over to my temporal workflow, I can see, hey, it's running. We detected, we analyzed, we plan repairs, we executed. I didn't do a human approval step here because I wanted to keep things moving, but you could do that there. Then I did a report with original data. This one's the one where I expect things to break because it will have the most conflicting data. I can see that it ran in the first attempt. One thing that happened quite often was this one would just fail and need temporal retries to make it work. totally worked that's cool um and so i did some command line output um these are my proposed orders it's looking pretty good which it should this first analysis agent doesn't have any conflicts so it should perform pretty well but let's look at the final um repair report i'm going to refresh it and see um so it's a little less confident than last time um It looks like it ordered extra inventory for some reason. So we'll have to look into that. And then we did get our order approval. So it's a little more verbose than the other report. And it looks like I'll go back to my planning report too and see what it was trying to plan. Yeah, this is fascinating. So one of the things. that I noticed, I'll just go back to my summary results here. This is pretty typical. When you give it too much information, it seems to get confused about what it did and what it already did and what it thinks it needs to do. What I found is it hallucinated that it'll need to order more inventory than it really should because I'm giving it conflicting inventory. I'm saying, here's the inventory. before my tools to repair had run. And then here's the inventory after. And then it gets all confused about like, well, which is the real inventory file? Because I intentionally injected confusing, conflicting poison context. It was also pretty common that it would be confused about the state of things and just say strange words. LLMs sometimes do that. A lot of low confidence in the right tool and a lot of hallucination with problems. it would say like these problems aren't fixed when they really were when i looked at what the the temporal workflow did and when i looked at um what the agent outputs were um and that again comes from the poison conflict so i'm going to try to get through this slide real quick and give you at least a little bit of time cornelia um generally We kind of talked about this a little bit, but generally it's a good idea if you're building agents with Temporal to have a workflow that handles orchestration of agents, human in the loop, and keep your durable shared state to use that to pass relevant state to activities for their input. Activities can also load their data, call read tools. Activities should do one agentic task well, and then they return important information to the workflow to be part of the permanent shared state. But don't include what isn't relevant for future agents. And the way you do this with temporal is it's all code. You pass inputs to the activity method or function and then return what you want to send back. uh someone said this to me and i really liked it i wrote it down um temporal makes it easy to fall into the pit of success so when you're building with temporal agents and you follow these guidelines it makes it pretty easy to manage context well because you just return back what you need if you're used to building with that programming model so i'm going to stop sharing now and let cornelia take over super thank you josh those are such cool demos um let me go ahead and i think i'm sharing my screen you're seeing things right josh yep okay cool So for our final scenario, we're going to really talk about that. And you even saw elements of it throughout Josh's demo. So agents that are running for a long time. So the conversation history and by conversation, I mean in that with, you know, big air quotes, the conversation history could in fact just be a whole bunch of LLM loops as well. But what we're doing is we're adding to the historical context and that context is going into the subsequent turns into the LLM. So that's really what we're talking about here. And really the solution to that is to do what we call history pruning or conversation history pruning. Pruning falls into a number of different categories. There's a number of different techniques that you can use. And the one that I'm going to show you here today is summarization, which is to basically take this long history, summarize it, and then use that summary to seed further conversation. And how do you do that summarization? Well, I think it's been in the industry. We've really found that using an LLM, not necessarily the same model. This is another one of those techniques that we're finding works well in this AI space is to keep the LLMs from suffering from confirmation bias is use different ones. So use maybe a different LLM. I'm not doing that in the demo here. I'm using the same model, but you could use a different model to do that summarization. So we take all of these little things that are here and we generate one block that has the summary and then we continue our conversation. The other thing that you could do is you could just trim off all this stuff. So first in, first out. And then there's a number of other techniques. And we're going to talk less about specifically what we're doing to prune that, but we're going to show you some techniques that are going to allow you to do the pruning and address these other concerns. Some of the other concerns that you have to be aware of is auditability. So that conversation history might be subject to compliance. audit requirements in your environment so you may not want to just throw away that old conversation history even though you don't want to confuse the LLMs with it anymore you also might want to address provenance so if you were to do a summary for example and then continue the conversation understanding what generated that summary what model generated it when did it generate it what exactly was the context that went in to that summarization so tracking that provenance also may be part of a compliance thing and then there's the mechanics of operations like how do you do that when do you do the history pruning if you're in the middle of doing a whole bunch of work are you going to stop that work while you do the history pruning we're going to see how that's addressed as well So I'm going to spend the vast majority of the rest of this, just like Josh did, in demo. But I'm going to come back and forth and show you some pictures. So as I was thinking about the example that we would want to use to demonstrate some of these techniques. i really kept coming back to conversation history so a chat interface and so in fact that's where i decided to try with a little twist so you can see here that i have a basic flow it allows the user to have some direction on what's coming what's happening in this agentic loop but the agentic loop it's really just a chat loop is going to the llm for some some some information it's adding that output from the LLM to the conversation history, then it asks the user for input, adds that input to the conversation history, and then loops again. Well, none of you want to watch me type, so I'll show you what I did to make this a lot faster. So I have no idea what happened there. It went, let's see, where did, we were on this slide. let me go ahead and bring you over to my sample so i am going to yes i'm in the basic loop so i'm going to show you what is happening here and i'm going to kick this off so here's the uh i was just checking things out let's actually run the agent so up here in the upper window what we have is we have the actual loop running so this is the agent that's running now the the thing that i i the the the little hitch that i threw into this is that i am simulating the user input so i'm actually leveraging an llm to act as the user because like i said you don't want to see me type so the way that this conversation started is that the user said hello the assistant said hey what's up And the user says, I'm doing well. And have you done anything interesting lately? And so you can see here that the assistant starts the conversation and starts to direct the conversation. They get to the point where they start giving lists. And if we scroll down, you can see here. that I tried very hard to get the LLM to act as a user. And I said things like, hey, don't go and repeat what you said before, but these LLMs are non-deterministic. So I think my demos are going to fail beautifully. And you'll notice that there's something interesting that happens here as we go along, which is... that here's what the the assistant said one particularly compelling aspect of responsible ai and it gives some lists and the user says in return one particularly compelling aspect of responsible ai is and so no matter what i did and i didn't spend a lot of time on the prompt engineering but i spent a bit of time trying to get the llm to act like a user but you have to realize that these models are trained to be helpful and to give kind of authoritative responses so at some point the user responses turned into started looking like agent responses as we went so i'm going to go ahead and tell this one to stop and we'll see that stop in just a moment for those of you who know temporal you of course know that my client here is just issuing a signal into the running loop. And I'll show you the code in just a second. But here's the basic code. So here's where I am allowing it to stop. We'll get to summarize and pause in just a minute. But there you can see that we're consulting the LLM. We're appending the message to the conversation history. We're simulating the user's response with an LLM. adding that to the conversation history that's all we've done so far with just that ability to stop it at the top so i'm going to go back to the slides very quickly and show you what the next thing is that we're going to do so i'm going to extend this demo by not only allowing the user to stop the execution, but we are going to allow the human user in this case. So we're going to watch the demo. And when it starts to get out of hand, I'm just going to issue a summarize command. And I'll show you what we do with that summary in just a moment. I think I accidentally keep hitting a different key here. So let me escape out of this and go back to my demo. I need to Check out a, I'm going to check out the summarizing. I'm going to stop my agent and restart it. And now I'm going to start it up again. And so we'll see. Oops. we're going to see the conversation start, and it's going to start in much the same way. Now, while that executes, I'm going to come over here and explain a little bit more. So I have new code here, and what you can see is that the basic code is the same. It's exactly the same. So I've got, I'm executing the agent, adding to the history, simulating the user, adding to the history. But now if we drill in into these commands that can happen up here, is that before I executed the stop command and that stopped the loop and it returned and it closed the workflow. In this particular case, I'm going to execute the summarize command and I'll come back to the code in just a moment, but let's see if it's time to summarize. So the assistant is talking about impacting of AI in the music industry. And sure enough, the user has gotten confused. So what you can see here is that I've gotten some confusion happening. If you remember, that was one of the problems that we had. The other thing that you'll probably notice is that it's taking longer and longer to get the responses. And that's one of the other downsides of excessively long conversation histories is that it slows everything down. So what I'm going to do now is I'm going to issue the summarize command. And the next time through the loop, what we're going to see is that it's going to take the entire conversation history and it's going to run it through another LLM. And I'll show you that code in just a moment. so we're in the middle of the loop so the assistant is answering the user is answering again you can see here that the user is getting quite prescriptive instead of inquisitive in spite of the fact that i ask it to and now you can see i'm continuing as new with the conversation summary and here's the summary that was generated by the llm so it looks like we're still mirroring each other's responses let's stop that And so now we can start again. And you can see that it's kind of reset the conversation. So I'm going to go ahead and stop this and come back over to the code here. So when we do the summarization, the first thing that I'm doing is I'm calling this helper function, summarize to seed. And that's right up here. And where did it go? Summarize to seed. So I've taken the input list, which is the conversation history, and I've run it through an LLM with the instructions of you're summarizing multi-turn chat. I return that summary. And when I get that summary back, what you can see here is that I'm issuing a command in temporal called continue as new. What continue as new does is it stops the current workflow run it ceases that and then it starts a new workflow run with exactly the same workflow id you as the developer are the one who's choosing the workflow ids we're not choosing it on the server side you choose that but what we're effectively doing here is we're continuing the workflow but we're creating a different workflow run let me show you what that looks like in the ui So coming over here, what we can see is, and I already stopped it, that this is the workflow that started. So this was the original workflow run when I first started it. You can see that each one of these create calls is going to the LLM. It's going to the, it's creating a response, and that's what we've called it here. Here you can see the user command. that issued the summarize so this is you can see right here that we're summarizing and it was already in the loop so it completed that loop but then it stopped this workflow so you can see here 2c2c3 it stopped that workflow run and created the new workflow run which is right here so now the workflow has continued and finally this is where you can see that i issued the stop command so pretty cool stuff what you can see there is that the old workflow is still preserved so all of that provenance all of that auditability is given to you here so you have a first class abstraction in temporal that assists you in dealing with these context pruning capabilities so the final thing that i want to do is i'm going to go ahead and get this started because we're running a little short on time i am going to check out git checkout i'm going to check out main which is our final version i'm going to restart my workflow and let's see if i can actually remember so we're going to start that workflow and we're going to let it run for a moment what we're doing in this particular case is this final demo which is In addition to the summarize, which we were giving a manual trigger for, is that we've added what we call a durable timer. So this durable timer, by the way, I'm going to go ahead and now that we've let this run for a little bit, is that I'm going to say pause. And what pause is doing is it's simulating the user stepping away from the terminal. So they've stepped away from the terminal, and if they're away from the terminal long enough, we're going to seize the opportunity to do the summarization. Now, I'm going to do something really interesting here, and I'm going to come up here, and I'm actually going to kill the workflow running. So the workflow is not running at all right now. So remember that I've got a durable timer. Let me come back to my slides. So what we're doing here now is that we're in this position where we're waiting to see whether the user comes back or not. If they come back within the first 60 seconds, we aren't going to take the time to summarize. We're going to continue the conversation. But if they are in fact idle for long enough, then we are going to have that summarization thing happen and then we'll continue the conversation. So let's take a look at what's happening in the workflow. So you can see here that we have a running workflow and we are in the middle of a create attempt. It looks like something is timing out. I was expecting to see my timer fire, so maybe I'm going to have a demo failure here as well. Let's see for a moment if this is going to come back. What I was hoping to see here was that we had a timer running. oh i suspect it's because let's go ahead and start the worker again now we're simulating the user stepping away we should see the timer in just a moment if we don't i'll chalk it up to the demo gods ah there we go so the timer has started what i can do now is that i can actually kill my worker just like i did before and i won't take the time to let this time out for just a moment and the timer is going to fire in a minute regardless of whether my worker is running or not when the worker does come back online it'll recognize that the timer has fired and it will execute the summarization and continue the conversation so we had a little snafu there but i think we recovered in in just a bit So that was our final demo that gives you a sense of how we can use these temporal abstractions to manage the conversation history. So with that, I do want to spend just a couple of minutes, Josh and I want to kind of close things out here in summarizing everything that we talked about. Josh showed you a demo and some techniques around scoping your microagents appropriately. We had that great question in the chat that we answered that addressed exactly that first point, that in temporal, we can use activities or we can use workflows as those microagents, so child workflows or activities themselves. And then, of course, we'll use workflows to orchestrate them. The second demo that Josh showed you is the right context at the right time. So in temporal, this means that you're going to have durability for your context in those workflows. We didn't have a chance to show you, but we could have killed the workflows and had them come back. And that's where you're going to have your durable context. But ephemeral context, context that you don't need to be part of the conversation history, that can all be happening in activities. If you want some of the information from those microagents as a part of the durable conversation history, then you simply return it from the activity and it becomes part of the workflow and the conversation history. You're going to source the context via activities and tools, and like I said, return the values. And then finally, we're using things like continuous new and durable timers to manage conversation histories. With that, I'm going to leave you with a number of resources. There's a great blog that I invite you to take a look at that starts to talk about some of these things. There is the code snippet. snippets that uh all of the demos that josh showed are here in the order repair demo and the demos that i showed are in the chat simulator repository if any of you want to issue prs to get the user to act more like a user that llm be grateful for those There is a poll we would love to hear from you about what topics you'd like to hear in the future, hear more about in the future. But speaking of future, November 6th, we've got our next AI-centric webinar here at Temporal, and it's going to be on Human in the Loop. So I'm betting that many of you are struggling with human in the loop and how to do that properly in your otherwise somewhat autonomous agents. Come join us on November 6th and we'll talk about those things as well. So with that, I'll go ahead and leave those QR codes up. And we have... Maybe just a couple of minutes. I know that we have some colleagues have been doing just a fantastic job answering questions in the Q&A and in the chat panel. But are there any other ones? If you want to like open your mic, temporal colleagues and and bring up any questions, Josh and I would be happy to answer those if we can. There's a question in Q&A that I thought you might want to answer, which is how do you handle summarization? potentially losing key details and then losing important context from Daniel Miller? Yeah, that's a great question, of course. So I think there's a number of things. One of the things that I didn't do in this very simple demo was that I didn't put any guardrails in place. So one of the techniques that you have to use when you're using LLMs to do any part of your process is you have to add guardrails around that. So you notice that I used an LLM to generate the summary. What you might do in this particular case is that you might use another LLM to generate a summary or maybe not generate the summary, but say, what are the key points that shouldn't be lost in this? So you can use an LLM to implement some guardrails as well. That is one of the most important techniques. So that's one thing. The other thing that you want to put into place anytime that you're using LLMs for anything, whether it's summarization or just being an active part of your agentic loop, is that you want to establish evals. That doesn't guarantee that you're never going to lose context in summarization, for example, but you want evals in place so that you can constantly be, well, evaluating how your llms are performing so that you can get alerts if they start going off the rails or you can use those evals even as a mechanism to help you evolve those guard rails so i hope that answers the question but those are a couple of techniques that aren't just specific to the summary but just in general these llms you can't always trust them and so you have to put um uh You need to double check them in a number of different ways. Awesome. What else? There's a ton of good questions that I'm trying to find. Yeah, so we've got... time maybe if you could pick just like one or maybe two more and then we'll try to follow up with with answers in our forums so an invite for those of you who are not already in the temporal forums or in the temporal community slack please join us there and we'll continue the conversation in both those places what else josh so what i noticed was did we use what other tools did we use uh lane chain lane graph um light lm and For my agents, it's basically temporal, some Python libraries to manage JSON, and light LLM. That's the tooling I used. Same. You want to talk about this? Yeah, same. There's a bit of a misnomer out in the industry that if you want to implement an agentic loop, you need to use a framework that has an agentic loop built in. You might have noticed that. In both cases, we had while loops in there. We had while true loops. The agentic frameworks are great, and we do, in fact, integrate with a number of them, including Pydantic AI and the OpenAI Agents SDK, and where we can bring durability to those. They're generally not durable otherwise. They just are libraries that run in a single process. And they do have some other abstractions, like guardrails and MCP tools and things like that. that but uh we are finding we um as temporalites tend to to just write in pure python code um we've also seen that in some of our customers who have started with frameworks and then realized that in fact they can have more control and do everything that they need to do just in a code first approach Do you have time for one more or do we want to call it? I think we can do one more. Okay. I love this question from Ayush. I hope I get your name right. It is about concurrency issues when dealing with context and multiple agents, considering we don't want to give too much context to one agent, especially when dealing with data consistency and transactions. So maybe a little bit about how Temporal helps with consistency and concurrency. Yeah, so let me take a slightly different angle and we'll come back to the consistency and concurrency as well, although it's going to be related. So there's something interesting that we're doing here, and you might have noticed it when Josh was doing his demos as well, which is that our workflows oftentimes are acting as a digital twin for something. they're acting as a digital twin for an invoice or an order or an order processing thing and so a workflow actually gives you some level of isolation and so the context of one conversation for example so if i had started another conversation in another window i would have had two separate workflows and so we wouldn't have had concurrency problems with that um i'll come back to to actual the actual process concurrency in just a moment but we wouldn't have had any conflict because all of the conversation history all of the things that were happening are happening in each one of those workflows independently so we don't end up with clashes across the thing The other way that we handle that concurrency is with our worker architecture. So when these workflows are running, I'm not just running a Python process that is just running a single workflow. I was actually running a worker. And that worker can take care of many, many, many different workflows. Each one of those workflows has their own scope and context. but I can have concurrency because I've got a multiplexing worker architecture. I mean, that's a short answer. Anything to add to that, Josh? No, I was just over here beaming like, this is a great answer. If you've ever worked with frameworks that have a shared context situation, it can be much more challenging, but temporals. Worker architecture enables massive scalability and context separation. Then the workflows give you really great memory isolation. I could talk about that for a really long time, but it's a great question and I'm glad they asked. You covered it well. I just had to gush a bit. Super, super. So I think that's a great question for us to close out on. I know that we went over. I still see quite a few of you stuck around to hear our rambling. Of course, we didn't make 45 minutes. I was pretty sure we weren't going to. Thank you so much for your patience. Thank you for sticking around. Thanks for joining us for today. Look forward to seeing you again in the future. And Josh, it's been a pleasure. Yeah, me too. Thank you, everybody. Take care, everyone.

Context Engineering is critical for making useful, production-ready agents.

In this webinar, we explain common context engineering challenges and how Temporal makes them simpler to solve, giving you practical strategies for building agents that are more reliable, focused, and adaptable in real-world use.

Key takeaways:
- Common context engineering struggles.
- Managing context with Temporal: providing context, gathering the right data, focusing agents with tools, managing memory.
- Frameworks for setting agent goals and keeping them on track.
- How to bring it all together in a live demo.

---

Temporal is the simple, scalable, open source way to write and run reliable cloud applications. 

Learn more
Blog: https://temporal.io/blog
How Temporal Works: https://temporal.io/how-temporal-works
Community Slack: https://temporal.io/slack

Developer resources
Docs: https://docs.temporal.io
Courses: https://learn.temporal.io/courses
Support forum: https://community.temporal.io