Build Hour: Agent Memory Patterns

Hi, everyone. Welcome back to another Build Hour. I'm Mikayla on the startup marketing team, and I'm here today with two members of our solution architecture team, Emery, live in the studio, and Brian, joining virtually to help address Q&A throughout the hour. Hi, I'm Emery. I work as a solution architect with OpenAI supporting digital native customers on building various AI use cases, including long-running AI agents. So today's topic is agent memory patterns, which is a very exciting topic and Emory and I's first ever build hour. So if you've been following along, we started with how to build agents from scratch using responses API, then moved into agent RFT and today exploring agent memory patterns. All of the sessions are up on our YouTube channel, so definitely check them out if you want to catch up or revisit earlier builds. Though the focus on the Build Hour is to empower you with the best practices, tools, and AI expertise to scale your company using OpenAI APIs and models. So for today's Build Hour, we'll start with an introduction to context engineering, the foundation for agent memory, and then Emory will walk through several live demos covering memory patterns like reshape and fit, isolate and route, and extract and retrieve. We'll end with best practices. resources and of course live q a on the right hand side of the screen you can drop questions into the q a box anytime during the session our team is monitoring both in the room and virtually to help answer throughout and we'll save a few for the end to go through live all right with that i'll hand it over to emory to kick things off thanks michaela hi everyone um so i'll start the first part of this um the session with context engineering definition so this next definition from andre carpotti i'll start by emphasizing that the context engineering is both an art and a science so it's art because it involves judgment so you have to decide what matters most at a given step of uh reasoning or action processes it's science because there are concrete patterns methods and measurable impacts to make context management more systematic and repeatable so i'll highlight that modern llms don't just perform based on the model quality but they perform based on the context you you give them in this slide i want to talk about different disciplines that comes together to basically present the context engineering so it is a broader discipline than any single technique like prompt engineering or retrieval so the diagram visual represents the ecosystem of context optimization layers that together shape what the model sees and understands so you see prompt engineering as a core principle, structured output reg, state and history management. Memory is also a crucial part. So using persistent or semi-persistent storage like files, databases, or memory tools to upload and retrieve key information. And all of this is contained inside the larger sphere of context engineering. So we can also connect these capabilities into different product capabilities. Here is a nice summarization slide. that talks about the core principles, like why it matters, because long-running and tool-heavy agents, bloat tokens, and degrade quality via poisoning, noise, and confusion, and bursting. We have three core strategies, as we discussed in the beginning, like reshape and fit to the context window, isolate and route the right amount of context to the right agent. and extract high-quality memories to retrieve in the right time. We also have prompt and tool hygiene as a core principle, so keeping system prompts lean, clear, and well-structured. Use a small canonical set of future examples and minimize overlapping tools and get the tool selection. And our goal at North Start is basically aiming for the smallest high-signal context that maximize the likelihood of the desired outcome. And then in this slide is where our transition from why context engineering matters to how to actually do it in practice. So I'll frame this as a toolkit of techniques. So these are not mutually exclusive. So most real world agent architectures combine multiple strategies depending on the use case and the context budget. So the first technique is reshape and fit. We can apply context trimming, compaction and summarization. The second one is isolate and route. We can offload context and tools to specific sub-agents with a selective handoff. And the last bucket is extract and retrieve. We can talk about memory extraction, state management, and memory retrieval in that last bucket. When we talk about context engineering, it's essential to distinguish between short-term and long-term memory because they solve very different problems. uh i think we can group uh the first two buckets as short-term memory which we also call as in session techniques and then the last bucket is long-term memory uh we call it cross-session so that means you can um collect different information from multiple sessions and you can retrieve back in in the next session or other sessions uh in in the future so short-term memory is all about like making the most of the context window uh during an active interaction and active conversation uh and then in contrast long-term memory is is about building continuity uh across across the the sessions cool so we we often get excited about how powerful our agents are becoming how am models are getting them better and better they can handle complex tasks uh route between tools they can plan multi-step workflows but underneath all that there is a there is a core bottleneck because context is is final so every piece of information we add to the prompt instructions conversation history tool outputs competes for a space in a fixed uh token budget and this is why uh it matters slide i want to make the problem concrete here so i'll frame it around before and after contrast so you see two conversations um what happens with our memory on the left and what happens with memory on the right. So on the left-hand side, the user started with the issues like Wi-Fi battery and overheating in IT troubleshooting agent. After many turns, the agent has forgotten the earlier context. It falls back to re-asking for information that the user already gave, right? But on the right-hand side, the agent remembers the original issues. Even after many turns, it can pick up the unresolved threat. It references previous actions like firmware update, background scene, which makes it feel intelligent and reliable. So this is such a stateful behavior, which is the foundation of a long running agent. Now I'll switch gears to failure mode. So we can group these failure modes into four categories. The first one is context burst. So you can imagine it as a sudden token spike in one of our multiple components to do the limited external control or increase Context conflict, if there's any contradictory instructions or information in your context. Context poisoning, if there's an incorrect information that enters the context and propagates over the terms. It can be via summaries or memory objects, state objects you're injecting into the context. And then finally, context noise. So you can imagine it as multiple tool definitions. or like way more, many tool definitions coming into your context at the same time. This can be redundant or overly similar items, so that can make a noise in the context. Here's a nice visualization of context burst in tool heavy workflows. So you'll see that there will be like a specific increase in one specific turn, and you'll be injecting a large amount of tool tokens here. And then the next one is context conflict. So we can easily visualize it here. So you can imagine in one of the turns, there is a specific tool call. And here in this tool call, you see that in the system instructions, they never issue a refund if warranty set is not active. But in the middle of the turn, you're also saying that it's eligible a refund for VIP customers. at the end of the turn and your agent is responding hey you know given your urgent travel i can issue a full refund so this is a nice visualization or example for the specific context conflict that can be coming from one of the tool results and the last one is context poisoning so you can imagine it as a hallucination or something inaccurate makes into the context in any step and propagates across different terms. So here we have a couple of possible pitfalls here. Lossy summarization edits can be causing this. If you're using a free-form node that accumulates over time, that can be contradicted. And then finally, older summaries overwrite the newer ones and you'll be basically causing any hallucinations because of the summary logic and you'll be injecting that. hallucination into the into the context and propagating over time cool now i'll stop sharing my screen and switch to the demo that i prepared for you and then i'll go over all of these some of these challenges um to show you like how it actually works in a real world scenario okay let me share my screen cool so Here, I prepared a demo app for this build hour. It is an IT troubleshooting agent for software solving issues, for issues related to both software and hardware. And this is a dual agent demo that lets you run two agents side by side. So backhand logic states inside the Next.js app, and I'll be using OpenAI Agents SDK. So we have two tools connected to that agent. One of them is get orders. Another one is get policy. So here I can start sending a message and saying hi to both of the agents. And then you'll see that both of the agents are responding to my message. And then here I can say, hey, my laptop fan is making weird noises while I'm playing games. Is it normal? So here you see that the configuration for both of the agents are same. They're the same model, same reasoning level. But I'm sending the same message. there is no memory configuration yet. So here you also see the context usage bars at the top. So you're seeing different type of components that is already in the context now. And I can say, hey, before that, I want to see my orders. My order number is one, two, three, four, five. And so now I'm expecting the model to make a tool call and show me. uh the orders i have so here you see that the order status the items i have and it's powered by a specific tool called so as you see over time um it will accumulating different tokens and different type of tokens here and in the context life cycles i'm visualizing what is happening under the hood across multiple turns so here you see that i have 84 tokens of system instructions my user input is increasing uh slowly uh but the core component here uh is agent output that will be generated by by a model cool um so this is a typical real world scenario so now i also want to showcase like how context burst is is happening so i can still uh start with high and i can say hey this time i'm having an overheating issue on my laptop And then the model is responding to my message, to my issue, basically. And then here it's telling me specific instructions, and it's asking me some questions to better understand what's happening. And then I can say, hey, thanks. Before that, I want to see the refund policy of my MacBook Pro 2014. So while I'm sending this message, I also want to... quickly show you the code and core concept of how it's working so it's powered by openai's agent sdk and here you see the agent definition so it's a customer support assistant i'm having specific instructions here that i'm adding i'm using different models here and also i can show you the system prompt really quickly instructions So it's basically, I'm saying that, hey, you're a customer support assistant for devices. And then I'm using that very slight prompting and instructions for that specific agent here. So let's go back to the response. So since I asked about the specific refund policy for my MacBook Pro 2014, it's made a tool call. called get refund and it's basically returning a specific um refund policy that i added before so here you see that in between turn two and turn three there is a specific spike um in the in the context window so in turn two i i had maybe around like 300 400 tokens but now i have more than 3 000 tokens because I am just dumping lots of information into the context. And then this is a nice example for context burst here. Instead of maybe just dumping all of this information into the context as a refund policy, I can be more careful about my tool definitions and tool outputs. to basically make a decision about like what should i inject into the context so maybe not all of these information are valuable but as you see that i'll be injecting lots of information into the context that's visualized here uh in this context life cycle tab cool now i'll stop sharing my screen and go back to to the deck so i'll continue uh with the next uh steps here okay Nice. So we talked about challenges and what's going on under the hood and a specific example about context first. So now let's talk about the solution, right? So the solution is basically managing context efficiently using different techniques such as trimming, compaction, state management, memories, and make the natural step beyond prompting. Again, this is also another visualization about different components in the context. So you see that across the times, it's increasing, the token counts are increasing, and these tokens can be coming from the system message, user message. Maybe you might be injecting memories, or maybe you might be injecting different type of specific tokens that can be added into your context. Here, I want to group. uh ai agents in terms of context profiles so we can group them into three categories so first one is rack heavy assistance so you can imagine reports policy qa agents um in these type of agents context is mostly dominated by retrieved knowledge and citations the second one is tool heavy workflows uh so context is mostly dominated by frequent tool calls and returned payloads and last one is conversational concierge so you can think about planning agents coaching agents and in this case context is mostly dominated by uh growing dialogue history there'll be lots of tokens in conversation history like assistant usage tokens that scales with uh with session lengths and then to better understand the the solution and uh the techniques we can go over what is fixed in our context and what is dynamic and in a variable So here you see different type of components, like usually system instructions, tool definitions, and examples, unless you're doing a RAC approach, is mostly static in the context. What is dynamic is tool results, retrieved knowledge, memories, and conversation history. So these are the nice examples for dynamic and static context and tokens. So you have a control on dynamic tokens, and you can... apply different techniques to control it efficiently. So I would like to start with prompting best practices to avoid context conflict here. You can also find it from our prompting guides and cookbooks. The first rule is being explicit and structured. We suggest you to use clear, direct language and specific enough to guide action. You should give room for planning and self-reflection. I think this is becoming more and more important with reasoning models like GPT-5. uh and you should avoid conflicts so keep the tool set small and non-overlapping don't use uh ambiguous definitions even if a human can pick a tool the model won't either so be careful with conflicting instructions and tool definitions so for context noise we talked about like many many tool definitions and many tools attached into the context as an example situation again you should be explicit and structured in your prompts More tools isn't always equal to better outcomes. So favor targeted tools with clear tool decision boundaries, and then return meaningful context from your tools. So in the demo, you show that specific example of context burst. So we suggest you to control basically what is the tool output, and then return high signal semantically useful fields, and prefer human readable. identifiers nice so now i'll switch gears to the engineering techniques and i'll start with the first one which is reshape and fit so the first technique here is context trimming so it is a pretty basic technique it basically means that dropping older turns while keeping the last uh end terms uh here in the turn and we have like limited context uh it's getting getting noisy there lots of information in the context. It can be coming from a tool, user message, or different sources. And there is higher likelihood of losing track because we are getting close to context limit. But once we trim the older conversations, older messages, now we have fresh context. It has better attention. And you'll see that it will also increase the latency that you're using. So it basically keeps the last end messages. and trips the previous older messages. And these are just some parameters and now we have control over in context streaming. The second technique is context compaction. It basically means just dropping tool calls or tool call results from the older terms while keeping the rest of the messages. So if you have a tool-heavy agent, you can consider these techniques. You'll see that your context will be mostly dominated by tool results. will be noisy there'll be maybe some context noise and lots of information coming from different tools and after compaction you'll see that a little bit fresh context uh better attention and and faster uh processing uh and you will also be keeping the tool placeholders uh intact after even after the context compaction here and a question you might have uh would be like okay how can i decide heuristics about trimming and compaction So here I can share a couple suggestions. So first, you can analyze your sessions. You can collect context snapshots from production or from your users. You can collect times down and dislike context to see what's going wrong there. Think about the average token size of a context. What type of task do you have in one session? Secondly, do not trim mid-turn. and break turn blocks. So a turn basically means that a user message and all the other message until the next user message, right? So if you just break or don't respect these turns, there will be higher likelihood of losing track. And then finally, don't wait to hit context window limits. So keep track of context allocation. You can set thresholds like 40% or 80%. So if you're getting closer to hitting the context window limits, these thresholds will help you to better understand, basically, when you should trigger some of these operations. You can control tool outputs, and you can also keep track of token saves. So these techniques are also really nice for cost, reducing cost purposes. uh and you can always keep track of like how much token you're saving while you're increasing the the overall capability of the of your agent and then the next technique we have is context summarization um so it basically means compressing prior messages into structured summaries and you can inject into the context history so here uh in turn n you see lots of messages noisy context again and you're basically keeping the last n messages and just summarizing or compressing the previous ones um so that you'll have fresh context uh better attention faster processing and at the end of the day you'll have a golden summary so this will be like a valuable information because you'll be basically compressing all of the valuable informations uh and at the end of the day it will have like a very dense um object that you can keep track of and they'll be also useful for you to better understand what happened in the conversation and here's a nice visualization uh in the context life cycle uh about the summarization so let's say you perform the summarization a specific turn so you'll see that you are compressing uh all the previous information and injecting it back uh into as into the context as a memory object so you see there is a new component um called as memory after the summarization performed next um so here here is a comparison of summarization versus trimming uh so there are different dimensions that you can consider while you're designing a memory pattern for your for your agent um you can see that in trimming uh you'll just keep last entrance and you'll be dropping the oldest ones so basically it will be pretty straightforward operation so it'll be very fast uh you know there is no no latency uh but the trade-off is that you you might be losing some information uh that is already there so i think this is the main main trade-off that we have uh it's it's really best for tool heavy ops and and short uh workflows and then in in summarizing you basically just keeping track of all the information so you are not throwing away any anything This can add a little bit of latency and cost because you'll be doing another summarization call to a model, but you'll be just collecting all the information. So you can think about an agent or use case. If you have multiple tasks in your long running agent that are independent from each other, you can definitely consider trimming because probably the trimmed or throw away information is not important for the agent for the next turns. But if you're collecting useful information across multiple terms and tasks are dependent to each other, then you can definitely consider summarization here. Nice. So now I'll stop sharing my screen and go back to the demo. I'll show you a couple examples of these techniques that we already covered here. Let me quickly share my screen. Nice. So here, let's go to configurations page on the demo. So for agent B, I want to enable trimming. And I can basically set max turns as three to trigger the trimming operation. I want to keep recent turns as three. So here, I can start again to test my agent and saying hi. So this time, I want to understand the refund policy that I want to check. Maybe I want to refund the laptop I just bought. I can say, hey, I want this refund policy for my MacBook about a month ago. And I want to understand what's happening here in terms of refunds. So if I'm eligible or if I'm not eligible. So the model is now making a tool call and calling the get refund tool here. As you see, it's returning that specific information. I have totally return window for returning that specific laptop. And I also want to check my order. So I changed my mind and I'm saying, hey, can we also check my order? My order number is one, two, three, four, five. And I want to see if it's on the way, if it's coming. So you see that the model is doing another tool call as get order. And now I want to switch gears. I can say, hey, thanks. I'm having an issue with the internet connection. So until this turn, you see that I have lots of tool tokens here in the context, in context lifecycle. So it's getting accumulating over turns. And in that specific turn, now it's telling me that, hey, let's sort it out. It's asking me a couple of questions about my device. And I can say, hey, I tried to load an internet page and still see 404 error. I basically share a couple important information about the situation. And you still see that across the turns, it's accumulating lots of tokens. And even agent output is increasing. OK, so let's say it's still happening on Safari. This is the last message probably that I want to share about the current situation. And I'm waiting to have more guidance and instructions here. And here you see that at the end of turn six, the context is trimmed. If I go back to here to visualize what's happening. So when I hit the turn six, you see that it trimmed the context. So it basically removed all these tool outputs and tool tokens. So now I have a fresh context. here. And then now I can continue talking about the same specific issue, or I can continue to talk about different information. Nice. So let's go back. And then here, I also want to show you how summarization works. So also the compaction works in a similar way as streaming. So here I can set compaction trigger. as for and keep recent recent turns too so you'll see that at the end of turn two it will be compacting and removing all these tool outputs and i'll have a fresh context similar to to the trimming approach but now i want to be a little bit more advanced here so here i want to enable summarization um i want to set the summarization trigger as five and i want to keep the the recent three turns so here i i clicked save and now i want to just see how it's summarizing all this information so here i'm sending hey i'm having internet connection again so this time i decided to share more and more information about my my situation so i can say where about this computer um what is is the model so i can say hey i have a 2014 macbook pro 14 inch and i live in the us But I bought it from Amsterdam. I received it from a battery chain service and just updated the OS version last week. And they asked me to update the OS version to MacOS Sequoia. So as you see, I'm sharing many information. And these informations are really available for an IT troubleshooting agent, right? And here, I can go back and say, OK, these are nice. Clarify the problem, what I need. from you and i can say hey i already tried hard reset after checking the faq docs but it didn't work so this is still in a form of memory because i'm sharing like which steps i tried which worked and and which didn't work i can go back um and continue talking with the agent so now it's reasoning itself and providing me like more detailed guidance uh and instructions for a specific tool to a macbook so I went back to my computer and I saw that the Wi-Fi icon is not active. And then I'm thinking maybe it's related to Wi-Fi or maybe it's related to a specific software issue. So as you see across the terms, it's getting more complex. The agent needs to reason. And the agent also needs to keep track of what's in its context and make sure that there is no burst there is no conflict um there is no poisoning and other type of failure modes it's telling me uh some specific steps and i can say hey i tried it already all right and i'm wondering if it's a specific software issue right um and then here i'm waiting for the response from from the agent so again i shared lots of information here uh lots of available information so now the agent knows my device where I bought this device, which steps I tried, and basically what type of steps I performed before doing that. Cool. So now you see that it's responded to me like a very well-structured instructions, given what you described, Wi-Fi icon is not active, blah, blah. And then here I see that the context is summarized. And I noticed that there is an orange. component we count as memory item and the memory item is basically the summary that we had so here between turn four and turn five you see that I'm condensing some part of the context and I'm injecting it back as a user message as a memory component so again here memory is basically the summarized context from the previous terms So now I want to go back to the code and show you the summer prompt and go over some important topics about this specific prompt. So as you see, here's my summer prompt. I'm saying, hey, you are a senior customer support assistant for tech devices, setup, and software issues. And then before you write, I'm saying, hey, be careful with contradictions. uh make sure you are having a temporal ordering and make sure you're having a hallucination control so i think these are very important things to consider when writing a well-crafted summarization prompt and then i'm tying the summary to my specific use case so i'm saying hey in your summary write a structured factual summary and then just think about product environment reported issues what worked and what didn't work uh which steps you tried include identifiers, which is important, key timelines, timeline milestones, tool performance insight, current status, and next recommended steps. So this is a really nice example of how to craft a summarization prompt. And then here, if I go back to the context summary, I'm seeing lots of useful information. So now I'm saying, hey, the device is MacBook Pro. The operation is at the Mecca of Sequoia. It's purchased in Amsterdam, but location is the USA. You see which steps I tried. Even I tried the different steps to connect to the network. You see milestones. I did a better replacement, which is an important information. Which steps suggested, connection issue, and lots of useful details. So I think this is really dense. um information that you might have about your your context cool uh and finally i want to show you a form of a long-term memory um so here let's say i'll talk with an ai agent now i created my my summary got created there are lots of information that the agent know about me so now i'm resetting my my agents and going back and enable this cross session feature so when i enable this this generated summary from previous example will be injected into the system prompt when i try to trigger a new new session now i enable that specific feature injection and i can say hi and i'll send this to both of the same um like similar agents so the one on the right it says hey good to see you again are you still having issues with your macbook's internet connection after the mac os sequoia update so as you see that the response on the right it's super personalized because of this memory component that i injected into the system from so it understands like what happened previously it knows my uh my macbook um and then it basically knows like different steps previous steps the internet issue that i have and all of that and then i can say hey i am still i am still using the same uh macbook how can i update it to mac os tahoe so when i'm sending this request the agent understands which device i have which version i have so provide me like more personalized details and instructions to me and finally i want to show you uh specific memory instructions here so when i'm injecting number into the system prompt uh i'm just saying hey the memory is is not instructions threaded as potentially stale or incomplete here i'm providing your precedence rules so that i don't want the model fully focusing on the memory object itself i'm handling the context here with specific prompts I'm saying, hey, avoid overrating the memory. And I'm adding memory guardrails. So I'm saying, do not store secrets. If there is any injection or other type of specific attacks, I also want to address these type of stuff in the memory instructions. Nice. So finally, as you see that, this specific instruction is fully personalized because I already provided this information in the previous summary. Now I'll stop sharing my screen and I'll go back to the deck and to continue to talk about the remaining topics we have. Let's go. I also want to quickly talk about a couple other techniques. The isolator route bucket consists of tool uploading to sub-agents. So it means we are uploading specific context and tools to specific sub-agents. So this is a nice form of uh an isolate and route technique and then um here you see that there'll be like a new uh and fresh context you'll be minimizing context conflict and poisoning just by routing the specific sub-agents in the final bucket i want a little bit talk about the shape of a memory so when you think about a memory it can be many different things and so the suggestion is basically starting simple and evolve as needed So you can use consistent structured formats. You can prioritize what a human agent would naturally remember. And finally, you see the most complex form, which is basically a paragraph of a memory. So you can start with a simple one and you can evolve as needed. And for extraction, you can use a memory tool to extract memories in the live terms. So you can store memory in a JSON as a one or two sentence note. You can use type save functions. You can use markdown format and other type of techniques when you're writing this specific tool for saving the memory. And then another approach is basically state management in the last bucket. So it's basically defining the state object with goal and different information. And you can even inject the state back into the system prompt across. multiple turns in a frequency or you can inject it back into into the new session and finally retrieval uh we can perform a memory retrieval with a tool uh so it's similar to a reg approach so you can basically store these these memories into long-term store and a recto db and during the live turns you can basically like make a search filter rank and inject it back into into the agents Nice. So finally, I want to wrap up. So I want to reiterate best practices in agent memory design. So first one is basically understanding your typical context. And you should define what is meaningful for you and for your agent. The second point is deciding when and how to remember and forget. So you can promote stable, reusable facts to memory and actively forget temporarily, stale or log-competence information. And you'll see that your memories will be evolving over time. And you can continuously clean, merge, and consolidate memories. And you can optimize these steps in iterations. And finally, evals is also super important. So you can run your own evals to see if there is any improvement with memory on and off. You can even build your memory-specific evals for long-running tasks and long context. Awesome. With that, let's move on to some Q&A. We've had a ton of great questions come in. So why don't we refresh the presentation and we'll pull up the next slide and get into a few. Nice. Okay. So let me go back to the Q&A session and jump into the questions we have. So let me quickly share it again. Nice. Okay. Yeah. Let's start with the first questions. Yeah, so there are any libraries or packages to work for context engineering. So this demo is built by using OpenAI's Agents SDK package and library. It gives you a really good flexibility to implement your own sessions. And in these sessions, you can easily implement like trimming, compaction, summarization, and all of that type of techniques easily here. So I see. many different libraries that are evolving really fast to basically make your life easier for context engineering. So as you see, we have too many techniques, and each technique has different parameters to tune. So I also see that there is an evolving part of all of these libraries. But I can suggest OpenAI Agents SDK as a starting point. to basically start implementing specific context engineering techniques and go from there. Next. Next one. So how do you evaluate or measure the memory feature is evolving, the memory is improving performance? So this is a really nice question. Yeah, after this session, you might think about, hey, I implemented. a specific memory approach, but I don't know if it's good or not, how it's performing well. So we can maybe split this into a couple portions. So first one is basically just running your regular evals with memory and without memory. So I think this is a really nice way to start thinking about if memory feature works or not. So if you have some specific eval metrics like completeness, like upwards, downwards, or that type of metrics, you can see if there's an increase or decrease or if there's any statistically significant uplift coming from the memory. And then maybe your evals might not be capturing well that type of memory-based boost or improvement, then I suggest you to think about more like memory-based evals. So what I mean by memory-based evals is basically evaluating the model on long-running tasks and long context. So if you are not hitting any context thresholds, maybe your agent doesn't need any of these memory improvements at all. So again, you can start with your core evals if you have already. And then secondly, you can start creating your own memory-based evals. So you can even evaluate the quality of the summary. you can evaluate the injection time you can evaluate the injection prompt so there are different ways to evaluate it but of course in most evals you also might need to prepare a golden data set first and think about maybe like a couple 50 examples like golden examples of a good summary or you can try different heuristics that i mentioned before to basically find the right balance of trimming uh and compacting so i think we can just group this into three different buckets first one is running your own evals to see see if there's an uplift second one is building memory specific evals and the third one is basically finding the right heuristics and parameters to apply in the in the context engineering techniques next one so should we use hierarchical context like entire project context for immediate task and context for immediate file edit in questions um so yes the question the answer is yes but it's mostly dependent on the use case so we also have a concept called memory scope so you can think about it as this memory scope as a global scope that means if you have a customer or user of your agent probably there are some information that you should always remember about that specific user maybe this user likes more friendly tones maybe this this user lives in the us so these are some examples for global memory But you can also have a scope based on the specific session. So let's say I wanna book a travel and then this time I prefer window seats because I wanna sleep. So this is also a nice example about the session scope and session memories. So I think this is a good practice to maybe separate these into two buckets and you can keep track of session memories with session scope. And over time, you can graduate session memories into global memories. And you can keep track of what is really important about the specific user. So in travel concierge example, if user is always saying, hey, this time I want window seat, maybe multiple times, and you can finally graduate that memory into global memories and keep track, keep it in agent's mind, basically, and remember that. for the next bookings. Nice. OK, so what strategies do you use to keep memory flashed or pruned so the agent doesn't become overloaded with stale aura? Yeah, this is another good question. And in the real world, you see that memories are evolving really fast. So after some time, you'll see that there are some memories that you need to prune and the agent needs to forget. So in that specific case, there are a couple of techniques to apply. So first of it is basically keeping a temporal text. Okay, I learned this memory from the user, but I learned it maybe like two months ago. So if you can keep track of these timestamps or temporal text, the model will understand what is old and what is new. So if I say I like... dogs if i said i like dogs like two months ago and today i say i like cats so you'll see that the model is going to understand my favorite animal now is maybe cats and it will override the memory with the right instructions so this also falls into a little bit to memory consolidation so how to prune stale memories how to basically update and overwrite the new ones into the existing ones so temporal text is one technique that you can apply The other one is you can use a way decay or basically a window function. And you can basically focus more on the recent memories. And you can basically downgrade the oldest ones. So it really depends on the nature of your use case. So if you think that what I said a year ago is not important for your agent, you can definitely prune these old ones and implement a weighted. average probably for all your memories but if you think that all of this memory is is equally important for your agent then you can consider maybe like memory consolidation and memory override with temporal text so we can talk about two different uh techniques um to manage the overloaded and stale uh memories nice okay so have you managed scaling agent memory systems when you have many users with individual and shared memory pools yeah this is also another good good example from real world um so once you see the memories are evolving over time and you'll see that you're collecting tons of memories from um from your users um so there are different ways to scale it i think the first path or first first decision criteria starts with if you are basically performing a retrieval or search-based long-term memory approach, or you're just using basically summarizing the context. So if it's the second one, that means you're just storing all of this information and persist it into a disk. So you can think about some scaling methods about data management, how to manage a large amount of memory notes as a text in a text format. Or you can basically think about scaling the first approach, which is basically you have to think about how to scale a search and retrieval system. You might be storing all of this information into a vector database. And then in this vector database, you can try to scaling storage. You can scale all these vectors, filtering, ranking system. and all of that. So I think the first bucket is mostly about this long-term memory. So we talked about memory as a tool. So if you can think about extracting memories with a tool and retrieving back during the live turns, probably this is the situation where you're going to hit this question about scaling for many users. In this case, you can think about scaling techniques for vector databases. You can use sharding. You can optimize your embeddings model probably if you're using a customized embedding model and you can basically optimize the retrieval process similar to a reg approach again the first one is mostly about scaling a retrieval system the second one is mostly about basically like data storage how to store specific data how to manage tons of information and sentences i think to wrap up we can basically put it into two buckets. One of them is scaling and optimizing a retrieval system. The second one is also making more efficient for storing and persistent in the disk. So this is also a common question that I hear from my customers. I think you can maybe follow like a pilot approach and you can turn on this new uh memory techniques for for for a subgroup of your users and you can think about okay how it's evolving over time maybe you'll see that most of the memories that your users are saying are pretty limited so think about this travel concierge agent so probably i'll just sharing my my memories about my seat preference maybe if you want to book a hotel i like maybe higher floors maybe i like the specific menu or breakfast uh so i think this is more limited type of groups uh type of memory memory possibilities i can say and that type of agent but you are if you're building a life coach or life coach agent so there are tons of memories that you need to remember uh about me my life uh and you'll see that these type of memories and memory pools are evolving really fast so yeah the third point is that try to understand the the evolution of memory and possibilities of memory in your ai agent so we have two examples here travel concierge memories and then life coach memories so yeah as you see in the second one you'll be collecting tons of information that is valuable for uh yeah for my life um and then my dreams my goals uh what i was thinking a month ago or a year ago so the second one is is mostly like super advanced and complex and sophisticated memorable that requires lots of scaling for sure okay and so yeah that was the end of the question the QA session probably yeah okay yeah and then we can just switch to resources so All right, this has been awesome. To wrap things up, we've linked a few great resources here, including the Context Engineering Cookbook, which was referenced, and the Context Summarization Cookbook, and our Agents Python SDK. I know we've gotten a lot of questions on, is this available in GitHub? So you can explore all of these links on the right, and the full BuildHour repo is available on GitHub. so good news we're likely going to squeeze one or two more of these in before the end of the year so keep an eye out on our build hours page linked here and um a big thank you all so much for tuning in and a big thanks to emery who's did an amazing job with this session yeah thanks everyone uh we hope you enjoyed uh this build hour on an agent memory patterns i know we covered like lots of different techniques uh lots of different information about memory how to think about memory, how to design memory. So overall, as you see, there are too many options. But the core idea is basically better understanding what your agent should remember and how it should remember and how it should forget. So you can think about these three things when you're designing your own agent memory. And this is still an evolving field. So you might see some new features coming. about memory overall. But yeah, I just wanted to show you like different design trade-offs and guide you with the best option. So finding the right balance between these techniques are usually like related to your specific use case. And then you can keep track of all of the news and cookbooks in the resources section. So I'll be also uploading this demo page. So demo application to our... build hours github uh and then yeah thank you for your time and thank you for for listening uh all of this yeah have a great rest of your day and we'll see you next time

AI agents don’t just reason — they remember. In this Build Hour, we deep-dive into context engineering techniques that enable agents to maintain short-term and long-term memory, personalize interactions, and operate reliably across long-running workflows.


Emre Okcular (Solutions Architect) covers:
• Why memory matters: stability, personalization, and long-running agent workflows
• Short-term memory patterns: Sessions, context trimming, compaction, summarization
• Long-term memory patterns: state objects, structured notes, memory-as-a-tool
• Architectures: token-aware sessions, state injection strategies, guardrails, and memory triggers
• Live demo: building an end-to-end agent with dynamic short and long term memory
• Best practices: avoiding context poisoning, context burst, context noise and context conflict.
• Live Q&A

👉 Context Engineering Cookbook: https://cookbook.openai.com/examples/agents_sdk/session_memory
👉 OpenAI Agents Python SDK: https://openai.github.io/openai-agents-python/
👉 Context Summarization with Realtime Cookbook: https://cookbook.openai.com/examples/context_summarization_with_realtime_api
👉 Follow along with the code repo: https://github.com/openai/build-hours
👉 Sign up for upcoming live Build Hours: https://webinar.openai.com/buildhours/

00:00 Context Engineering
10:44 Context Lifecycle Demo
20:13 Context Engineering Techniques
26:49 Reshape + Fit Demo 
39:16 Conclusion
42:45 Q&A