Context engineering for AI agents

Context is everything when we're talking about building reliable and effective AI agents. Knowing how to manage the context windows of our AI agents is an essential skill for anyone who is building AI agents for real users in production or just fun demos you make trying to impress your friends and family. I can't be the only one. So it's no surprise that the term context engineering, which focuses on how we make AI agents more reliable, has gained popularity when talking about building AI agents. So for this lesson of AI agents for beginners, we will take you from concept to code on answering the questions of what is context engineering, what are good context engineering strategies, and common context engineering issues solutions for them this video follows along with a written lesson that goes more in-depth into this topic as well as including translations you can find a link to this above this video and below in the description you can also meet other learners ask questions and attend workshops about building AI agents in our dedicated discord channel here so let's get started by looking at what is context engineering When I think about context engineering, I think about grocery shopping, which I assume most people who eat food has done at least once. Because each AI agent has its own grocery basket, which in technical terms is the context window. And like a real grocery store, everything we put in that grocery basket has a cost. For large language models, that cost is tokens. or basically the amount of things the LLM can process in the context window. Context engineering is the practice and focus on managing what goes into our context window by taking a systematic approach to ensuring our AI agent has the right information at the right time to complete a user's task. So to continue with our grocery store analogy, this is making sure we are putting in healthy food that we want to eat and it's good for us. and not filling the basket with food that is necessarily healthy or bad for us. Now, this is putting context engineering in very simple terms. But when we later look at the strategies around good context engineering and the code examples, you can get a more in-depth understanding. But a question you might be asking is, what about prompt engineering or retrieval augmented generation? Well, these techniques contribute to the different types of context that ultimately build reliable agents, which is the goal of context engineering. These are the items that are in our agent's basket, for example. Context types include instructions, which are prompts, system messages, and few-shot examples to help guide the AI agent. This information is static, but context engineering also focuses on the dynamic data that our AI agent uses. Knowledge, which includes a RAG system, that retrieves information from different databases and data stores, as well as long and short-term memories of our AI agent. Tools, which can be external functions called by our AI agent via APIs, MCP servers, or just in our code base somewhere. Conversation history, which is the chat interactions between the user and agent over time that will grow as the conversation continues. And lastly, user preferences. which allows our AI agent to respond to the likes and dislikes of the user over time. But now that we have a look at what context engineering is, let's look at some of the best practices on doing it effectively. Like so many things in life, good context engineering starts with a good plan. So let's first look at what the steps are in creating a good context plan. The first step is to define clear results of your AI agent. What will the world look like when the AI agent is done with this task? And I don't mean that your AI agent should change the world here, but maybe it might. This is more to say, what is the change or information that the agent will have done for the user after completing the requested task? The second step is to map the context. This should answer the question of, what information does the AI agent need to complete this task? This should be where you map out the required context. Again, this could be instructions, tools, and knowledge that the AI agent will need to complete the task. And the last step is to create the context pipelines. This answers the question of how will the agent get this information? This covers creating the connections between the knowledge and tools, which could be creating a RAG system or connecting to an MCP server. That covers more of the planning. But let's look at some practical strategies for context engineering. Context engineering is all about managing the context window of the AI agent in a more active way. The written course describes this in six different ways. So check out this when you have a chance. But we will only look at three of these in this video. The first is the agent scratchpad. This is where an agent either uses a separate file or an object available at runtime that allows the agent to take notes about the current session. The advantage of this is that this information can stay out of the context window and is only called when needed, which you can instruct the agent to do. The next one is memories. We have a whole lesson on managing agentic memory in this course, but to cover it in simple terms, while the scratch bag can be used for managing the context in a single session, Memories is used for multiple sessions across time. This is great for managing things like user preferences across chat sessions and adding personalization to your AI agent. Lastly is compressing context. This can be done by either summarizing the context as it grows and keep relevant information or pruning, which just removes older messages over time. We will look at how this works in a code sample. So let's take a look now. Okay, so we're here at our code example. It is called chat summarization in the GitHub repo, which again, you can find at the link above this video as well below in the description. We're going to actually do two things here. So you're going to get a code example two for the price of one, continuing our grocery store analogy. One, we're going to look at this concept of agent scratch pad. So we're basically going to make this agent Create a scratch pad. It's going to be a very basic markdown file of just basically it's the task that it's being completed and what's kind of going on in the conversation. It's going to track a user preference kind of object. Second thing we're going to do is also just chat history summarization. So it's actually something built into semantic kernel and allows us to designate what we want the chat history summarized. give it a target in terms of what it means uh to summarize that into like how many conversation points are going to be including the summarization and when it should actually do that so let's look at this so you're going to do all the imports of the required packages as normal and also load load in our environment variables which are here again if this is your first time looking at this course or the code samples we have a setup chapter in the github repo to get you started there The next thing we're actually gonna do is to create this Agent Scratchpad plugin. So plugins in this example is basically how Semantic Chrono uses kind of different tool calls. And in this case, we're actually going to create this Agent Scratchpad MD file. So when you run this, you should be able to see when the Scratchpad is being made. And then we're actually gonna have it so that every time there's an update, uh you know based on the user's travel present preferences or completed task uh the agent should be adding that there again this is going to then take some of that information out of the context window we don't want to lose that if we summarize certain things so just kind of keeping that information uh valuable uh for the agent to retrieve later on then we're going to use this chat history reducer so we have two kind of main parameters here that we want to set which is the reducer target count so this is exactly uh what it says in terms of the target number of messages we want to keep after reduction does it always get this in terms of um you know making sure that we still have the same context but is the target that we want to kind of hit and then the reducer threshold is when this actually should happen so after we get 15 messages in this case we want to then start looking at reducing the chat history again allowing us to take away some of that out of that context window or agent's basket and giving it a nice summarization. Then we're going to just kind of define this, what service we're going to use, the chat service that we've already designated. And then we're going to do a lot of these things because this is for educational purposes. We're going to print out a lot of these things. You don't have to do that in a production environment or an actual application, but just to kind of show you what we've already set. then we have this agent that we've defined again this is the instruction so this is still part of context engineering basically telling it to when to add anything to the agent scratch pad and when to also read that uh if there's anything like the in this case we're going to have this agent do some travel planning so if there's any changes in the location but we still want uh the agent to know the preferences of the user so it doesn't have to continue to ask that we want that in the scratch pad and want it to be retrieved so we have this kind of very basic planning process of read the scratch pad ask about preferences and nothing there update the scratch pad and then create any itineraries and also keep a log of terms of like the completed tasks that you have Then we have these helper functions. Again, this is going to be really kind of showing you more than just telling you on how this kind of chat reduction history summarization works. So we're just going to look at the amount of tokens we've had and we just want to get a clear, like hopefully beautiful graph of what the effect is of all these things that we've kind of put into place. You're going to skip that. You'll see it in action later on when I show you kind of the printout of these conversations. And then we have this user input. Again, I wanted to show, you can play around with this and change things, but obviously I want to show since this is kind of about, you know, managing context that we have at least something necessary in the context, especially when we're talking about function calls and things like that. That's obviously all that information goes into the context window. So that's going to be really important, but this is just going to be kind of a basic chat around, you know, planning a nice vacation there. And then we're going to look at this. So what we'll do is actually run this thread, the thread that comes in through the history reducer. So the chat history is actually going through the history summarization instead of just a thread before being handled. We're going to also do some token tracking, which is going to be really nice to show the amount of tokens in the context window. Once it gets reduced, how many tokens are in there. And this is also really important to talk about. Not only context engineering, but even cost savings, which context engineering kind of lends itself to as well. So if we go down into the conversation now, you know, this user is asking about planning a vacation. We have the agent as expected to ask a little bit more information around destinations, their preferences, dates, things like that. They go in there, the user goes in, you know, have this conversation. And again, keeping a good count here of how many messages are in the thread or in the context window. And then we continue on, you know, we're doing all this planning. The actual user changes the location. First it was Bali, and then now it's Greek Island. So really, really good choices all around. And then you'll see once we hit this thread, 21 messages. over our threshold the length which is over 15 we actually get this history reduced kind of alert again this is just for educational purposes you don't have to have this in your application and then we not we actually see uh what the summary looks like here in this case uh the user's planning vacation uh you know we show what that would have been recommend uh recommended as well as the uh you know the budget uh and just a little summary of the conversation which is actually good because now kind of you know switch uh the topic a little bit more into like now that's nothing about the weather uh and the conversation goes so on and so forth in terms of um you know reducing uh this sort of things and then looking at that but what i wanted to show you and what i've added here into the notebook just to kind of show you exactly what we mean around um what this reduction is doing is you can see as the conversation goes on, we have 481 tokens, 830, we hit 1,000, nearly 2,000, and then we do some reduction. And then we get down to 1,501. And again, you can look at this as just... purely on a cost saving perspective but we're also making sure uh the context is being well managed and not necessarily having information that might go over this is clearly not even getting close to the context window sizes that some of these models have but something to consider and something to think about when we talk about context engineering so we see the amount of tokens and then we keep going as the conversation and as you know you can add more inputs here and see how's it goes going um and how many times the context gets reduced. So then you'll see this reduction amount, and then you'll see the token saved. Nice little calculation there at 23.5. So when we had it before and after, you can see what the actions are and the actual effect of that. And then in this agent scratchpad, so you'll see also in the markdown file, but we've got this printed out here as well, where we'll have this. So I have new user preferences. So we have the destination. This shows the timestamps because it's getting updated every time. And then also the completed task. So in this case, we created itinerary for beach vacation in Bali. So that sounds nice. And this, again, showing all of these concepts of chat summarization and 18th Scratchpad in action. OK, so we just looked at the code examples. And do check it out to run and change it on your side. The truth is that managing context doesn't always go as planned, unfortunately. Let's look at two ways that this will happen and the solutions for it. Again, in the written lesson, we have covered four of these topics, which is at the link above this video. The first one we'll look at is context poisoning. This happens when false information, mainly generated by the LLM itself, enters the context and the agent continues to use that information. This is particularly bad when we talk about long-term memory as well. An example of this could be in our travel use case that the LLM creates a hotel that doesn't actually exist when trying to book hotels for a user. This could continue to lead to errors over time if not removed from the context. A solution to this is to validate information before it enters the context of the agent. So if the agent uses an API to book hotels in real time, that information should be validated before adding it into the context of the agent another issue might come up is context confusion this happens when too much information that is not needed is added to the context a common area of when this occurs is in tool calling when the agent has so many tools to choose from and ends up choosing the wrong one due to the number of tools or the descriptions of overlapping a solution to this is to use the tool loadout management. This is where you use a vector database to distort the descriptions of the tools and have the agent select only the most relevant tools through RAG for each specific task or query. This acts as a filter and gives better results because only the most relevant tools are retrieved. So this ends the video lesson, but check out the code samples so you can run them on your own and the written lesson of this course in the link in the description for more information. And I hope to see you in the dedicated Discord channel here as well as the next lesson of AI Agents for Beginners.

Prompts are just the start of guiding our AI Agents. Context is everything. In this lesson, you will learn how to identify, collect, and serve the right information and tools to your AI Agents at the right time. You will get a practical understanding of the growing field of context engineering and how it applies to building reliable AI Agents. 

➡️ Find the sample code at https://aka.ms/ai-agents-beginners

Find the full "AI Agents for Beginners" Course and code samples here ➡️ https://aka.ms/ai-agents-beginners 

Join the Discord to get your questions answered here ➡️  https://aka.ms/ai-agents/discord 

Chapters:
00:00 - Introduction  
01:20 - What Is context engineering?  
02:30 - The Different Types of Context 
03:56 - Context Engineering Best Practices 
06:35 - Code Example 
14:40 - Common Context Engineering Issues and Solutions