Context Engineering Explained: How to Build Reliable AI Agents

Hi everyone. Welcome back to VectorLab. Today, we're going to talk about context engineering with Emre. Emre, can you start by presenting yourself? Yeah. Hi everyone. I'm Emre. I'm a solutions architect at OpenAI. I support different customers to build AI agents. So I've been focusing on context engineering and memory in the past couple of months. And I'm excited to share more on that topic today. Yeah, that's a very hot topic. So can you start by going back to basics maybe and just start with a definition of what context engineering is and why does it matter? Why do we want to talk about it today? Yeah, definitely. So I think in short, context engineering is the art and science of filling the context window with just the right amount of information for the next step. So it sounds like a very simple definition, but actually it's really hard, and that's why I say like art and science, because the context of an AI agent is changing every step. Different stuff that we add to context, then we remove the context, shrink the context when we hit the context window limits. So basically the core technique is just managing it in a better way so that our agent is not getting lost. In this tech, I'm going to cover context engineering and different techniques to manage the context efficiently. And then here is content, the topics that I'm going to cover. We also have a demo to basically illustrate how context works in a typical agent and how we can apply, reshape, and fit type of techniques in the demo itself. Cool. And here's a definition that I shared in the beginning. Again, it's art and science. There are different techniques which are scientific. And then there are different techniques which we consider maybe like a little bit of art, art of context here. But again, this is a nice starting point. And I think it summarizes like what we have to do in context management. The goal is basically just simple. Let's put the right information, right amount of tokens for the next step, basically. And then as you see, this is a multidisciplinary field, I can say, the context engineering. So it has different techniques to perform. You see a prompt engineering, which is one of the core piece here. React technique, state, and history management. Memory is one of them. Structured output is also another technique. So you can see all of these combinations as contributing to context management techniques. And then the right balance of using these type of techniques, I think this is the best recipe, I can say. You can also see that Context engineering is like a natural evolution of prompt engineering. So I'll also talk about that in the next slide. And then core principles. I think this is also a nice summary. It matters because long-running tool-heavy agents blow tokens, then they create quality over time. So context might be poisoning. There might be some noise and confusion. And then we have three strategies to avoid that. Reshape and fit to context window. Isolate and route to the right amount of context to the right agent. and extract high-quality memories to retrieve in the right time. And of course, prompt and tool hygiene is still important. So keeping system prompts lean, clear, well-structured, using small set of examples, minimizing overlapping tool definitions is also pretty important. And then the North Star is always aiming to basically maximizing the smallest high signal context for the likelihood of the desired outcome here as you see we have three buckets in terms of techniques reshape and fit so you can you can see it as a context operations mostly like trimming the context compacting the context or summarizing it isolate and route is a second technique so we are basically handing off some part of the context to a sub-agent or multiple sub-agents. And then we have extract and retrieve. So it's similar to the reg type of techniques here. You can perform memory extraction in real time, state management, and also memory retrieval can be considered under that bucket. So if you want to just split short-term and long-term memory, I think this is the right definition. So short-term basically means in-session memory. Like whatever you do, it basically stays in a single session, and when you close that session, it's gone. And then cross-session is basically long-term memory. That means if I said something, hey, I like dogs in one session, and in the next session, this memory will be remembered. So that's also the definition of cross-session. uh type of memory i think we get those questions a lot from customers like how do i make sure that i build a system that can remember my preferences the preferences of my customers i love the distinction between the short-term memory and the long-term memory and all those techniques brought into a slide of what can you do to to address this problem yeah definitely i also hear in many times from my customers too and these type of terms can be confusing sometimes what is short like what is long basically but i think there's a nice categorization to start with uh and then you can think about basically if i need something in my session or if i need to personalize my agent cross session so i think this is a nice starting point and question so you think you see basically more and more both being used or one over the other Or it depends on the use case? Yeah, good question. I think it depends on the use case mostly. Some agents require short-term memory and it's enough. But some of the agents, let's say life coaches, like AI therapists, that requires basically a continuation. For that type of use cases, long-term is definitely a better fit because memories are also evolving over time. It's remembering something maybe it needs to forget. But I think as a starting point, short-term memory is definitely a good way to start. and then you can evolve if it's needed. But you can definitely see good gains, even just implementing a short-term memory implementation. Then you can move to long-term if it's needed. Good point you made there, that if you have long-term memory, then you have to have some tactics on how you handle even that memory, right? So for how long can this be long? So how long am I keeping this memory? Exactly. Yeah, remembering and forgetting is definitely like very close terms. And then you'll notice that you'll have too many memories at some point and you have to consolidate it. You have to basically just prune or delete the old memories. You have to override it. So these are also like very sophisticated techniques in memory lifecycle. so i'll also talk about that in the next slides cool uh and then yeah so we have a problem here okay we want to build smart agents personalized agents that remembers and forgets but the problem is context is a finite resource whose effectiveness diminishes uh over over the turns so we know models grow more capable uh the real challenge is is shifting from like simply writing perfect prompts to carefully selecting what information is important to put it in the context. This is also a nice illustration to show that the difference between memory-of and memory-on type of agents. So LLMs get lost in long-running multi-term conversations, as you know. So the left one basically is asking the same question again, hey, how can I help you? Even I told like many details during the conversation, but the right one is basically the magical moment for an AI agent. It remembers what was the problem. It says, hey, great, we solved this problem. Let's pick up what's remaining. We discussed about this earlier. So based on your profile, this is your device. So I think this is the target and the goal that if you want to build a really personalized and cutting edge AI agent. Cool. And in the context, we have four failure modes. So probably you'll recognize one of these if you're building an AI agent. So context burst is one of them. It means suddenly increasing the token, the total amount of tokens in the context. Context conflict, if there are two instructions that contradicts each other, it might confuse the agent. Context poisoning is basically having a hallucination or incorrect information somewhere in the context. This can be in a system problem. This can be in a... user message or tool output or somewhere else. And context noise is basically adding many, many similar stuff in the context. I think tool definition is a good example for that. If you don't have clear tool boundaries, you'll see there is a little bit of noise in your context. So these are the four failure modes for agent builders that we see from the field. So this is everything that can go wrong. Do you think we'll ever get to a point where... because models are getting better and better and the context windows are becoming bigger and bigger, that we won't have the problems of at least reaching the limit of the context window? Or do you think we'll always have it because we want to bid systems that are more and more performant? Yeah, good question. Yeah, context windows can get longer and longer. But this attention mechanism in LLMs is basically designed in a way that it has a limited budget. the token in the beginning of the context and at the end of the context has different impact to model, I can say. So even if we have longer windows, I think there will be definitely like a need for optimizing the context in a better way because it's also important for latency and it's important for cost. So instead of dumping everything to context because you have a large context window, might not be the right way to think about it. uh since yeah if in the future probably will continue work with llms with attention mechanisms i expect that these type of techniques will still be to be useful uh for that cool um yeah here's a nice example in context burst um so on the x-axis you see number of turns number of back and forth uh conversations and y-axis is the context window allocation so you see around turn 20 there is a tool call and suddenly you're just dumping a large a number of tokens as tool results into the context window. And now you're hitting almost maybe 100% of context allocation in the next term. That means your context is huge now. There are tons of information maybe you don't need, but you're just bursting your context just by adding tons of tokens from a tool result. And in your experience... Sorry, Maria, good question. Okay, so I was thinking that... What are the best practice or things that we need to track in our systems and AI agents that can help us see what type of context burst we are having? So then we can fix it. Exactly. So you're definitely right. So you have to have a way to basically understand what your context is. What's the typical context that you have? Is it dominated by... tool results or it's mostly dominated by agent output or user messages and so you should definitely analyze your context maybe you can pick random samples from from past conversations and understand what is in your context and see how it's dominated by so this definitely requires lots of context analytics I can say you can you can maybe categorize this is this a new thing now context analytics Actually, yeah, it's a good keyword. But definitely it's important because otherwise it's just like a text file, right? You don't know what's in it. So you can see what type of tokens you have, the average length, if there's only one task you're solving in one session. So there are many questions to ask about your context. But yeah, you're definitely right. So you have to find a way to basically analyze what's in your context to understand the issue. And here in this particular example you're showing, we can see it's bloating because of the tool results. It's obviously a case-by-case basis. But in your experience, do you think this is the main cause in general? Is it the first thing that people should try and think of? when they hit their context window, do you have any tips because you've seen so many cases or do you think it really depends on, I imagine like the type of agentic workflow that you have, right? If you have a RAG workflow, maybe it's like the tool call, but maybe if it's something else, you know, do you think it's generally because of the tool calls that it bloats? I can say yes. So this is a common failure because usually tool calls is not governed because It's basically an API call and most of the time you don't care about what's the output of the API and you're not checking anything. So there is no guardrails at that checkpoint. So definitely what I see from the field is that two results and controlling two results is very important. So make sure it's not super long. Make sure you're not dumping like hundreds of pages of PDF file as a retrieved file. many chunks so I can definitely say tool results is the way to start a starting point to think about like how to how to avoid the tool burst so context burst so any for example if we think about mcp and mcp tools or is this gonna play into these tool results that we're having here yes that's for mcp tools and retrieved knowledge rag is also I think in this category it's basically also like a tool call so if you are just dumping all the chunks you retrieved uh this is also a typical failure mode uh we can count as a context burst so then in the open ai world do you guys have a way of tracing and finding at each step or cool sorry tool call what are the requests and responses so that then we can figure out the categorization you have had here yes in responses api we have a way to keep track of what is uh like a tool results full output and what's the like models generated response based on that tool so you can definitely keep track of these three type of messages uh in in the context history cool uh and then the second failure mode is context conflict um so there can be a conflict conflicting information between system prompt and the tool output as in this example so you can say in the system prompt hey uh your customer service agent never issue a refund if warrant status is not active and then there are turns or conversations back and forth and somewhere in the middle there's a tool call uh for checking warranty status and it says policies is like it's eligible refund for vip customers and then there's another message from a system that says hey understood given your urgent travel i can issue a full refund right now so you see that there are three conflicting uh information uh somewhere in your context and the model is getting a little bit confused because you're seeing something in the system front but there's something conflicting with it uh in the in the tool um tool results so this is it Typical example, maybe you might have it in your context already, but you're not aware of it. You should definitely check if there's any specific conflict or not. You can basically just paste the whole context to check GPT and ask about, hey, is there a heart conflict? And it will tell you, hey, there's a conflict, you can fix it like that. So this can be also a nice way to basically detect and fix the conflicts in the context. And context poisoning is basically having a hallucination somewhere in the context. So in this example, if there's hallucinations somewhere in the turns and you're summarizing these turns, this hallucination will be basically in your context summary. So that means it will be propagated across the turns. You'll keep talking with the agent and there's a hallucinated context summary somewhere and it will summarize again and again. So you'll have that specific hallucination in the context that will basically impact your agent in a bad way. Cool. uh now i'll switch my screen to to the demo yeah so this is a dual agent demo um there are two agents side by side and i'm sending a message uh to both of the i'm sending the same message to both of the agents at the same time but the difference is one of them has has memory uh and context engineering techniques enabled and one of them and then the use case is basically an i2 troubleshooting agent so i'm trying to solve a problem about them the software or device that I already have. So here, let me switch back to that screen. So I basically asked, hey, I need to solve a problem here. And then I can start saying something like, hey, my laptop fan is making noises while playing games. Is it normal? And then the agent is going to respond based on its system instructions and tools it already have. And then here I can say, hey, before that, I want to see my order. My order number is this. So there's also like specific tools attached to that agent, right? And then here in the context lifecycle, so you see different context categories for both of the agents. So we have system instructions. We have... 84 tokens as system instruction this is mostly fixed and over time we're increasing the tools uh agent output uh and also the the user messages uh here cool and then you see that like it made a tool call it showed hey get order here is your orders uh and all the other stuff yeah so we talked about a lot of uh problems and now let's talk about solutions and techniques to to think about it and the solution is actually simple like managing context effectively using different techniques as the next natural step beyond prompt engineering. So I already shared these type of visualizations about what's in the context. And then you ask questions about like what is the common failure or if it's long-term or short-term where it's different for one type of agent or other type of agent. I think these are three categories that we can group agents based on the context domination. So first one is rag-heavy analysts. You can imagine a rec agent that generates reports, policy, and QAs. The second bucket is tool-heavy workflows. So you can imagine automation agents, operations agents, or CRM agents in that category. So context is mostly dominated by frequent tool calls and return to payloads. And finally, we have conversational concierge like planning, life coaching, or therapists or companions is in that category. And in that context is mostly dominated by growing conversation history like user message and also the assistant message here. And then one question I had that nothing it fits nice like what you had here. So my question was, are these techniques or this context management context management something important or useful for chat agents? Or you've seen this being useful also for kind of more automated agents running behind the scenes? taking an action, let's say. Yeah, it's useful for both of them. So it's also useful for chatbot type of experiences if you have tools attached to that chatbot. And it's definitely also useful for like some like offline agents or background running agents too. So it's important for both of them because both of them has these similar context issues in just like a little bit. maybe with slight differences in terms of the content. Nice. Yeah, so before thinking about the solutions, we have to understand what we can fix or what we cannot fix or what is frozen, basically. So we have static components in the context, which is system instructions, tool definitions and examples, mostly. And then we have dynamic part, which is changing based on the conversation, basically. So tool results. is something that we have control on, retrieved knowledge, memories. It's up to us what to add, when to add, and how to add into the context. We have conversation history. Basically, it's purely dependent on undeterministic messages, like assistant messages, and also the user messages here. To start with, prompting definitely helps. I just want to... just um reiterate on the typical best practices for uh in terms of prompt engineering here so being explicit and structured is still important using clear direct language uh giving room for planning and self-reflection so it's also pretty important for reasoning models i think after uh five five one and five two are in this category uh avoid micromanaging adding every single step hey if this is that do that and that type of very strict instructions is not might be the optimal way to to add into the context and then avoiding conflicts just make sure your system prompt is not conflicting in itself right keep the tool set small non-overlapping tools and no no on no ambiguous instructions in the context so this is the first step like so if you feel like there's a problem of these problems like you feel this is often a step that people overlook maybe to just going back to prompting exactly yeah this is definitely a starting point uh and even you think that you have perfect prompt you might notice there might be some slight conflicts so it's always better to ask to check gpt hey everything is okay and you can even upload the prompting guide from models and paste your system prompt and make sure that, hey, what do you think? How can I improve it? So there's always a room for improvement in the system prompts, and there might be some hidden conflicts that you missed. But yeah, definitely, it's like a right starting point, and it can be maybe like a low-hanging fruit to solve the issue. Yeah, and same for the noise. Make sure you have the right set of tools, right tool definitions, clear tool decision boundaries. and then you just return meaningful context from the tool so maria asked a question about how to how to understand uh what's in the tool output make sure you're checking your tools consistently your apis are working there is no um issues maybe there's a bug in your system that returns like tons of tons of tokens and you're not aware of it that's also i think comes back to context analytics uh like hey maybe in this turn you're injecting like tons of unrelated information right uh so that also relates to uh that specific failure modes so as we discuss all of these it just comes to my mind how important testing is at this point right you have all of these components all of these modules like imagine if you just miss one tool that returns all of these contacts and blows off all your contacts right testing evaluate reasons like it's really important it is yeah so it definitely and it's it's not easy to keep track of all of this you're right maybe one turn in one conversation there was an error somewhere and you didn't control the tool output uh i think there should be some mechanisms to detect it maybe you can set some thresholds uh in terms of total tokens that's returned from from a tool i think that's i saw it from the field too that can be maybe like primitive type of approaches that you can set some limits. Hey, if there is more than 10,000 tokens in the output, let's trim it. Just send the first 10,000 tokens, for example. But you're right that even detecting it can be hard and tricky. Cool. And then let's start with the techniques that we discussed in the beginning of the slide. So first one is reshape and fit bucket. And the simplest one to start with is context trimming. So it basically means that dropping older turns while keeping the last end turns. So you just throw it away, the older messages, I can say, older turns, and you just keep the last one and continue to work on the agent in the next turns, basically. So this is pretty simple. There is no latency issues. You are not using any other LLM calls. So this can be a good starting point. The one... The disadvantage of this is basically you just throw away some context that might be useful for your agent. So if you have an agent that has multiple tasks and it's long running, you have long context, obviously. And then first task is not related to the second task. I think that is a good example for using context trimming because you can just trim away and it's not going to impact because there's nothing meaningful for the next task. So you can think about what your task is, what your agent does. So this might be a good solution for you if there is not related information in the context. And the second one is compaction. Dropping specific type of messages in the context. So tool results is basically a good starting point for that type of techniques. So in turn end, you have tons of messages. Your context is full. You have tool call, tool results. And then in compaction, you can just drop. these specific type of messages, and you basically compact your context, and your agent can continue to work on that. So after compaction, you'll have fresh context, you'll have better attentions, you'll see it's going to process it faster, and you can even add some tool placeholders to say, hey, there was a tool, but it's dropped, to make sure that you're not breaking the chain of thought of the agent. And heuristics, so we talk about analytics, right? how to understand basically like when to trim, when to compact the context. And then here are some starting points for you. Analyze your sessions. Context analytics is here. Collect context snapshots. You can especially collect the thumbs down examples. So if there's a session that your user doesn't like, these are also good targets for that analytics. Check your average token size of context. Check your tasks per session, average token size of a turn. These are all like good. metrics to keep track of and then do not trim your context between um do not break your context in the middle of the turn so a turn basically means that a user message and then everything until the next user message to detect them with tool call assistant message and all of that or meaning multiple tool calls so make sure you are not breaking that block uh and then you can keep track of your context allocation so if it's 40% or 80%, these are like good thresholds to do something basically about your context. You don't have to wait until you hit the 100% of context allocation. But I love that we have so many engineering tips on how to... you know have lower latency better performance and very often i feel we really overemphasize model benchmarks you know how good is this model and then you have this new model everybody says it's a better model because you know it brought us benchmarks but actually if you really care about optimizing for costs or for latency or performance etc sometimes you're better off like doing all of this you know or using all of these techniques to try and improve like your kpis rather than really just obsessing over you know like comparing models and how good they are and i feel this i agree yeah 100 so even we have same models for the next years there are tons of optimization uh opportunities here in the context layer right so you can improve your agents or like really uh um significantly right just just optimizing your context uh just managing context window adding uh memory states and all of that definitely i think this is really important layer optimization layer to focus on even models are getting better and better there's definitely like tons of things to do in the context layer yeah because people talk about pricing etc on this lower price but then for example if you hit the context window and you invalidate your cash you're going to lose money. And so, as you say, maybe if you spend a bit more time like engineering this and working on all this, that you could actually save a lot of money without having to change for a cheaper model. Exactly, yeah. Maybe you're dumping tons of stuff that you don't need to your context all the time. That's definitely a good way to optimize the cost and pricing. And also latency is also important. So you'll see that latency is going to increase because you're processing less amount of tokens. And then finally, I want to mention that long context window doesn't necessarily mean it's like a better intelligence or better models, because maybe you're just dumping in lots of unrelated stuff into context with that long context window. I think that can be also maybe like a common pitfall to think, hey, I have long context, 1 million token, 2 million, I can add everything. So that might not be the right way to think about it. But yeah, that's a great point that I'll see from the field. So one question I had was, how much of this is actually up to the user to implement versus abstract? You mean user like the user who is using the agent? The user who builds the system. Who builds the system, yeah. I think these types of techniques are mostly in the responsibilities of the agent builder, like an AI engineer, most of the times, I can say, because models are similar in terms of functionalities mostly. And then AI engineers should design and think about all of these types of different techniques, art and science based on the use case. So it's, there's no like one size. i'm trying to see if there is any abstraction on top of this for the builder to implement or it is the manual to think of different architectures and patterns on the context yeah so in terms of abstractions so agent frameworks has different features for managing context uh open source tools uh like land graph blank chain has different abstractions and openai agents sdk has different abstractions to basically play around with the context operations, but it's still evolving. So I can say that we have like a full suite of context engineering primitives to use for AI builders as of today, but you can definitely implement. trimming, compaction, summarization, all of other techniques by yourself using any framework. But I see that it's still evolving. So maybe in the future, we'll see better tools or primitives for AI developers to manage the context more and more efficiently. But yeah, this is definitely an evolving area in the field. Cool. Yeah, finally, summarization is another technique. It basically means like summarizing all the previous turns. uh using a summarization prompt and summarization called and injecting back into the system uh for for the next turn and then after summarization you'll see that your your conversation story will be shrinking uh and then you'll continue you know talking to your agent because you'll have a dense and summarized information and high high high signal details basically as a summary uh in your context so here the summary shows us memory after you've summarized. Exactly. Because summary is also a form of a memory because it's something you remember from the previous turns. Yeah. So you're basically adding in memory tokens and just shrinking or summarizing the conversation history. Here's a nice comparison. I'm going to use summarization versus trimming. There are different pros and cons based on your use case. Again, if you have multiple tasks that's related to each other, let's say in the first task, I have a problem. with my device and the second one, I want to refund that device, let's say. So in that case, the summarization makes more sense because you have different information in the context that you want to keep track of and you don't want to throw it away. Cool. And then, yeah, let me share again the demo to show you how this specific... um technique works in reshape and fit nice so in the configuration here uh i'm going to enable some specific memory controls here so before doing that actually let me show you like how context burst uh actually actually works here in this in this chat interface so i can say hi uh and then continue with a specific issue i i'm having an overheating issue on my laptop here But before that, I want to see a specific, maybe a refund policy about my computer, right? So it's telling me, hey, let's sort it out. First, I need more information. And I can say, hey, thanks. Before that, I want to see a refund policy for my MacBook Pro 2014. And then once I asked about the refund policy, it has a tool attached. And it's going to make a tool call to return like a long... refund policy for that specific case. So you see that my tool call is too long and it's returning like tons of information. And suddenly there are like tool tokens in my context. So you see that there is a significant increase from one turn to another one. I'm just dumping in like more than 2000 tool tokens. And this is also like typical illustration, an example for the tool burst. And then now let me enable some techniques here. And I can enable first the trimming, and I can set max turns as six and keep recent turns as three. So that means once I hit the six turns, it's going to trigger trimming, and then it's going to keep the last three turns in the context. And then I just saved it. So I can start with high again. And then I can say, hey, I want to see the refund policy for my MacBook Pro again. So this will return the same tool result about the refund policy to make sure that I have all these details coming from that specific API call. In the context lifecycle, you see that we are just starting. And then we have fixed system instructions as a fixed tool constantly. just discuss and then i can ask hey can we also check my order my order number is this this is also another tool call that we we can um that the model can basically check anytime in the in the context as you see i'm just jumping from one task to another one and then i can say hey thanks i'm having an issue with the internet connection that that's something different So this can be a typical scenario because in real-world examples and use cases, people are asking tons of different questions. This customer has too many problems. Exactly. Yeah. And then I can say, hey, I tried to load an Internet page and still see 404 errors. So the issue is getting longer and longer. There are tons of true calls in the context. In the sixth turn, let's see. So here we have at turn five. And I can say, hey, it's happening on Safari here. And then we're going to hit turn six once I send this. And then the A to B, as you know, there's the memory enabled. Memory configuration is enabled for that. So I'm expecting it to basically trim that session and see. here yeah and then i can basically ask something else in this uh session consider okay thanks yeah so here let me go back to configuration and here max turn is six and keep recent is this is three and the trimming is enabled yeah so it should be triggered in the in the system in the um yeah in the context life cycle here and then yeah as you see the context is trimmed here and it's going to continue with the less amount of tokens because i just throw away the the old um basically uh all turns in in the context so this one the agent b is has more like fresh context i can say we just drop the old information we don't need and then we can keep talking uh with the agent and you'll see it's not going to get lost but this one uh will have tons of different tokens in the context and it can get lost in the next turns. Amazing. And then, yeah, go ahead. You said amazing. I think I love that it's so visual because it's easy to understand how and why it's important to manage the context window because you lose information, but then you can start fresh again, which means you can do more turns. Yeah. And then we also have different implementations here for compacting. This is pretty similar. So you set your compaction trigger. If you hit turn four, it's going to keep the recent turns and just drop the tool results and specific messages. Can you remind what's the difference again between compaction and summarization? Is compaction just about tools? Yes. So compaction is just about tools. You just drop tool result type of messages in the context. Summarization here is about just summarizing the whole conversation. and you don't lose any any information and you just create like a dense and high value summary and inject it back into into the system so then there's another model called behind the scenes for the summary yes so the summary will be like super important in the next turns because it will be super dense it will have details about your device what was the issue what's solved and what's not resolved um and then you'll just add it into your context and you'll see the impact of this in the next turns uh in terms of efficiency but have you defined also what you kind of need into the summary like what a good summary look like exactly yes so that's a really good point so let me uh go back to the deck um and show you like the some tips tips and tricks for the the summarization um here Yes. So summarization is pretty important. You're right. So we should not use like a generic prompt to basically, hey, summarize this, right? We have to provide like some sort of details in the summarization prompt and it makes the difference. So let me share my screen again. The only downside is that it may not interpret the right way. So you need to make sure that it does. Exactly. Yeah. Compared to, I guess, trimming that you may lose. information or introduce some misinterpretation, I guess. Yes, definitely. Okay. Yeah. So summarization prompt is pretty important and it should be super related to your use case. So you should tailor the compression prompt to your use case. So in this case, the IT troubleshooting agent, it has different details in terms of device, software. what I bought, where I live, location. And these are all important and part of the memory, basically, to solve the task. And then here you see we have milestones, like highlight important events in the conversation. You should check the contradiction so the summary shouldn't be contradicting by itself. Timestamps is important and temporal details. Chunking, you should organize it in a way with titles and lists. Tool performance insights, guidance, hallucination control. many details and and here you see a typical summarization prompt you see hey your senior customer support assistant uh compressed the earlier conversations into a precise reusable snapshot for future turns and here you see for that use case we are just listing product and environment reported issue a single sentence of problem steps tried identifiers next recommended steps so this is a well-crafted summarization prompt for it troubleshooting agent and you see that all of these details are pretty important to keep it in the summary. Makes sense. Yeah. Yeah, the second technique was isolator route. So in this technique, we are basically just handing off the context and tools to specialized sub-agents. So it's also a typical technique in multi-agent architectures. Here you see on the left-hand side, we have one agent, we have tons of tools connected, and it's trying to solve the same problem. But on the right-hand side, we have an agent basically orchestrator agent and we have also two sub agents that we hand off some part of the context and then for example like ticketing and refund tool is attached to refund policy agent because refund is usually requires lots of reasoning you have to generate lots of reasoning tokens maybe there's like a refund policy pdf files that can increase your context a lot and then what we do here is we just solve that task in that sub agent and we just receive the the solution or basically just a conclusion from that agent to add it back into the context and then in this technique you'll see that we'll have fresh context we minimize the context conflict and poisoning and reasoning is also improved because each sub-agent is working on a specific subtask here and so here this is what you do when you use the agent sdk typically exactly agent sdk you can you can quickly implement this with sub-agent architecture and you can attach specific tools to sub-agents, and you can easily manage the handoffs. So you can hand off the context, solve it, and take it back this conclusion to the orchestrator agent. So is there at any point in time that all the context passes through this main orchestrator? Yes. Yes, okay. Yes. So in some agents, you need to pass the full context because the sub-agent needs to know what's happening since the beginning. So you can think about some maybe complex decisions to make in that sub-agent. But yeah, it happens. Some agent builders, they prefer to pass the full context over the sub-agents. But again, there is no actually like efficiency in terms of context entering, but... Maybe that sub-agent requires all the details, all the conversations and tool results since the beginning. It's really related to your use case. So if that sub-agent... Yeah. So here on the left, so the agent receives all the context and manages this, whatever it's going to happen afterwards. On the right, will all the context come again to the agent and the agent will just... hand it off or it won't. I'm asking that because there's some new research happening on this kind of recursive language models, RLMs, if you heard of it, which is the model itself handles the context, actually never takes all the context into its own context and then hand this off to different scripts, LLMs. small agents so is this that or at some point we'll handle on the context and then it's gonna hand it off yeah so actually there is not like a full context on the right scenario because it's handing off one task with a handoff note to a sub-agent so imagine like some set of messages uh forwarded to the first sub-agent and then the second task is forwarded to the refund policy agent so there is there is not any in time there's like a full bulk context on the right architecture so you can imagine that handing off a task with some handoff message instructions you can say hey your your orchestrator agents if there's any tasks related to orders or logistics make sure you're handing it off the order ops agent so you hand it off and solve it and take the conclusion back so it's basically how orchestrator agents work and therefore there is no like a bulk context in any time on the right architecture. Cool. Yeah, the final technique we have is extract and retrieve. So this is also similar to RAG techniques retrieved augmented generation in AI agents, which was very popular in the past years. But before that, I want to talk about a little bit of a memory. talk about context engineering, memory is also pretty important. And these are like very close topics to each other. And memory is a hot topic these days. As Lara said, it's still evolving. We're still thinking about what is the memory or what is the right way to implement it with machines. Even human memory is not purely solved. We have our short-term memories, long-term memories that work seamlessly. But since we don't understand it super well, there are some challenges to implement it with machines. So here in the shape of memory, we can think about different versions on the complexity axis. So a memory can be, hey, there is an issue, which is internet connection, and it's resolved or not. And the second form of memory can be a typical markdown list. uh with product reported issue steps uh tried and identifiers uh with timeline muscle or a memory can be a paragraph uh this is also a form of a memory because it tells about lots of details about my me as a user what's my issue is what i'm doing what what the agent tried what worked what didn't work so memory is is a very uh populated definition depending on the use case depending on the specific task. And yeah, there are tons of details you can add to a memory. So our guidance here is that starting simple in your AI agent and evolve as needed. You can use very consistent and structured formats like maybe a JSON or even maybe like a markdown list here. You can prioritize what a human agent would naturally remember. So this is really important. A good approach is just to asking a human agent, hey, okay, you solved this task. But what did you remember? What was in your short-term memory? Can you tell me what you remember about that user at the end of the session? And you can just note all of these features and you can just create your own memory structure based on that. And I'll always look for a balance between verbosity and structure. You should not be too verbose or you should not be too concise because you're going to miss some details. So there is definitely like a balance between these two. And you can add some memory guardrails. if your memory object is too long or conflicting or there's a sign number or prompt injection. So memory guardrails is also pretty important in that type of memory design. And the first technique here is the extraction. So you're basically using a tool to take notes or extract memories. You can say, hey, persist a structured troubleshooting memory to nodes.amd for later retrieval. abc that type of details edit environment details so this is basically taking notes in the live terms and storing it into into a file maybe a local file or into a json object how do you handle knowing when you should save a memory and when not for like customer preferences. Because in the case of an AI concierge, for example, let's assume I'm going skiing and I want to get extra luggage for my skis. And then the concierge remembers, Lara likes to have extra luggage for the skis. But then you want the concierge to be smart enough not to suggest that I take my skis with me over the summer, for example. Or, you know, maybe sometimes... user preference, they can change. I have New Year's resolution. I don't want to eat snacks. You know what I mean? And so I wonder how to handle that, you know, like deciding when you should save and not. Like, do you then ask an LLM to evaluate if that memory should be saved or not? Yeah, that is a common challenge. And there are different techniques to do that. So with the current mainline models, without any fine tuning, just zero shot techniques and prompting. you can basically write a well-crafted prompt to define what is important to remember and what is the scope of a memory so as you said i have skis for this trip but for the next trip i'm going to hawaii and i don't need skis right so you should add specific examples and instructions to basically tell the model what is um like a one-off memory or what is a global memory basically so you can write like instructions, and you can give examples. Hey, in this case, don't remember this, blah, blah. So it's really dependent to the use case. Got it. For travel concierge agent, you should define, hey, you know, there's maybe session-based memories. There's a global memories. Maybe I like to sit in an aisle seat all the time. Maybe this time I want to sleep and I want to have like a window seat. So you should define and write a well-structured tool descriptions and prompts to basically tune that over time. And second way is basically just tuning methods. You can tune a model for better tool calling and note taking. But I think even with the zero shot and scaffolding techniques, you can get to somewhere with tuning that tool definition. So I also have a question on as you build a system and as you iterate and evaluate the system, how would the fact that I'm not holding enough in memory will manifest is it gonna be not accurate enough like how will i know that is the memory that i have to go and check store more yes so you you should build your own memory evals to to understand that so you you can run your own ai evals right maybe you have completeness i don't know you have maybe you have different metrics with llm as a judge type of evals and you can enable memory and see if there's a difference first like With memory and without memory, maybe it's not going to help your evals at all. Maybe you don't need memory that much for your use case, right? And then as a second step is basically building your own memory eval. So you have to eval the quality of a memory, the quality of injection, also the consolidation quality. Am I forgetting the right memory or am I losing any information? So there are different steps to basically evaluate. and you also have to build a golden memory data set, which is hard to build over time. Basically, the right way to approach that question, Maria is basically just building a memory evals with different dimensions. Cool. Yes, the second technique here is a state management for context personalization. This can be also a good starting point for agent builders. You basically keep track of a state object across turns. You have your goal. um you have your customer details you have milestones uh in this json and during the turns you can inject or some details or notes into that json or you can read from that state object across the the turns you can inject it into your system prompt or you can just use specific fields uh depending on the the task and use case so this can be also like a nice starting point for a well-defined and narrow use case Now, finally, retrieval, it's a typical RAG-based approach. So you can write a tool to understand when to retrieve that memory. And you can do an embedding search. You can filter, rank, and inject that specific memory in the live turns. So this is the third option. There is a challenge in retrieval-based memories because you have to tune when to retrieve, how to retrieve, what is a good candidate. Over time, you'll see very similar memories in your vector store. But yeah, this can be also another way to do it depending on your use case. And this is the third technique that we have in that bucket. It's not easy to do because even ChatGPT, when I asked it what it knows about me, it was really random what it knew. So this is not an easy problem to solve, like select the right... Yeah, that's a very hard problem. Yeah. TLDR of this is memory. AI memory is hard. Because you see, there are tons of things to think about it. You can generalize it. It's super dependent to use cases. But yeah, so there are definitely some challenges. But I think there's also good research coming about memory and personalization in the future. I think this is a good starting point for agent builders. So hopefully over the next years, we'll see more improvements in terms of... these dev tooling primitives on the model layer side, on the tool side, and all of that. Amazing. Cool. Yeah, I know that was lots of information. Yeah, that was fun. That's the end of this deck. So let me stop sharing my screen. and then let me know if you have any other questions about context engineering and memory maybe one question where i get very confused very fast like in the context of you know you have different types of multi-agent architectures and sometimes i find it a bit confusing should you choose to have for example with the agent is here should you have agent with an orchestrator agent with a handoff to other agents or do you should you have other agents available as tools so first like the difference between the two and then how context management differs for each like architecture and do you have any recommendations because i try to always recommend the most simplest you know architecture because that's always what's going to work better like hopefully like one agent with lots of tools well not tons of tools but some tools But what's your recommendation on this? Yes. So this is a common question, architectural question that we hear from the field, like one fat agent versus multi-agentic system, right? So there are different pros and cons and different questions to ask to find the right fit for your use case. So one of them is number of tools. and modularity and then once you make your decision based on these trade-offs between one fat agent versus multiple sub agents and then context engineering is becoming more important as you said and so for for managing context in sub agents and multi-agent architectures you have to again think about your tools and your use case first so if you have a task that requires lots of tokens to solve these tokens can be coming from tools these tokens can be coming from a policy refund policy file with uh 10 20 pages of pdf and then you should think about forwarding or slicing that part of context and handing it off to a sub-agent because this will create lots of noise lots of tokens in the orchestrator agent context uh and then it's definitely better to just solve it on its own isolated space and take it back so this is the one common thing that you should think about like hey my task is too hard or maybe my task requires 20 tools attached. And I don't want to attach these 20 tools into Orchestrator Agent to increase the total number of tools. So these are the two decisions that you should think about it. And I think multi-agent is definitely going to help for that type of high reasoning or high token usage tasks. And of course, modularity is important. You'll have more room to route and classify with Orchestrator Agent. So this is also the third advantage to think about it. Thank you. Thank you. Thank you. Yeah, that was really interesting. Thank you for listening. I hope it was helpful. Very helpful. And yeah, let's keep in touch. Thank you.

In this episode of VectorLab, we dive deep into context engineering with Emre Okcular, Solutions Architect at OpenAI. As AI agents become more powerful, long-running, and tool-heavy, managing context effectively has become one of the most critical—and overlooked—engineering challenges.

We explore what context engineering actually means, why large context windows alone don’t solve the problem, and how poor context management leads to degraded quality, higher costs, and slower latency. Emre breaks down real-world failure modes like context burst, conflict, poisoning, and noise, then walks through practical techniques to fix them.

Through live demos and concrete architectures, we cover how to trim, compact, summarize, route, and retrieve context intelligently—plus how to think about short-term vs long-term memory, multi-agent orchestration, and when memory helps (or hurts). This is a must-watch for anyone building production-grade AI agents, copilots, or autonomous workflows.

00:00 – Introduction to context engineering
00:39 – What context engineering is (and why it matters)
02:10 – Context engineering as the evolution of prompt engineering
03:45 – Core principles and why agents degrade over time
05:10 – Short-term vs long-term memory explained
06:20 – When agents should remember—and forget
08:30 – Four context failure modes (burst, conflict, poisoning, noise)
10:40 – Tool calls as the #1 cause of context explosion
12:00 – Context analytics: how to see what’s actually in your context
14:00 – Conflict detection and hallucination propagation
16:00 – Live demo: agent with vs without context engineering
19:10 – RAG-heavy vs tool-heavy vs conversational agents
21:30 – Prompt and tool hygiene best practices
24:00 – Context trimming: when and how to drop turns
26:30 – Compaction vs summarization (and trade-offs)
32:00 – Designing high-quality summarization prompts
41:30 – Isolate & route: multi-agent orchestration patterns
46:30 – Memory design: structure, guardrails, and lifecycle
49:00 – When to store user preferences (and when not to)
51:00 – Evaluating memory quality with Evals
55:00 – One “fat” agent vs multi-agent systems
57:00 – Final thoughts and takeaways

#ContextEngineering #AIAgents #AgentArchitecture #LLM #PromptEngineering #RAG #MemoryManagement #OpenAI #AIEngineering #VectorLab