Crafting Intelligent Agents with Context Engineering - Carly Richmond - NDC London 2026

All right, good morning everybody. How are we feeling? So we can tell who's had their coffee very quickly from that. It's all right, it's my job at 9am in the morning to wake you up. I will do my best. So my name is Carly and today I want to talk to you about context engineering. So a quick show of hands, who's actually heard of context engineering? Okay, so we're maybe split half and half. So by the time we exit this room today, everyone is going to know what context engineering is. They're going to know what the practices are within it that we can leverage to make stateful and intelligent agents. And we're going to talk about why we need it as well, leveraging a lovely little demo that we'll get into later on. So if you haven't met me before, hello, morning, it's really nice to meet you. My name is Carly Richmond. I live in London, but I'm based, I'm originally from Glasgow in Scotland, not Aberdeen. And I work at Elastic. I lead the developer advocacy team with Endeavoril. That'll be four years in April, upcoming anniversary. And before that, I worked in banking as a software engineer for 10 years. Now, I know we're all on different socials. Everyone's all over the place. If you scan that QR code and you want to connect, you can find me where you are, whichever platform it is, because in being in DevRel, I collect them like Pokemon these days. But also, I will be at the Elastic booth if you want to come by and say hi, if you don't get time to answer your question or you want to dive into the topic a bit deeper. So it would be rude not to start with the definition, wake everybody up at nine o'clock in the morning. So we need to understand what context engineering is. And what it is, is the idea of assembling the information within an LLM's context window to allow us to build stateful intelligent agents. So this is one of the definitions I like that comes from the Sessions in Memory paper published by Google last year. Now, do we all know what context window is? Raise your hands if you do. most of us. So for those whose hands are very much stuck to their sides, the context window in simple terms is the number of tokens that an LLM can process all at once. So just like us, if you bamboozle us and put in so much different information, we're not going to remember it all. And as this particular illustration kind of demonstrates, we can have the situation potentially where there's too much information that doesn't fit in the context window, and it means logically the LLM's not going to use it. because it's you know basically out of bounds is the way i'd like to think about it now obviously we've seen with the enhancements with various models that context windows are increasing number of parameters are increasing so one of the questions i commonly get is well with these ever increasing context windows why do i care about filling an overflow Well, to be honest, if you throw everything at a particular model, not only are you running the risk of overflow, you're running the risk of not actually getting the right response back, particularly if you provide poisoned information, information that's incorrect, or even in some instances, information that's contradictory. So it's the same thing. We're trying to kind of communicate with each other. If we give each other contradictory pieces of information, we don't know how to process this. And these models are very much in the same, operating in the same regard. Now, one of the things that people always talk about with regards to LLMs is tokens and costs. That's one of the reasons we need to keep the number of tokens down is it's about money. And that is indeed true. That's part of it. But part of it, as I kind of alluded to when we talked about poisoning and other things, is that LLMs make stuff up and hallucinate. Who has had an LLM spit out something in the last few months that they've found to be incorrect or wrong? See how everyone's hand is up because they do and they do it for very innocent reasons So one reason can be down to the knowledge base on which they're trained on if you have a model that is trained to a particular cutoff point So for example November of 2025 and then I start asking about all the things that happened in the news over the last week They've been bombarding me with notifications. It's not going to know the answer It's not going to be able to tell me because the answer is not there But it can also be that it doesn't have access to other forms of information. So if we are going back to our organisations and being asked to build agentic systems for potentially external facing customers or even internal customers as well, they won't have access to that proprietary information that we have within our databases because they've crawled over public sources such as the internet. So you can be limited in that regard of knowledge as well. The second thing is with your choice of model, depending on which model you pick, which will be less of an issue with foundational models, to be honest, you have this construct known as overfitting. So within LLMs, exactly like other machine learning models, they have loads of parameters and weights and different values. And if you're trying to get it to do a particular task that maybe it's not. designed for or for the parameters of which are too tight you're going to end up with inconsistent or incorrect answers. The third thing is down to biases in the data set not just the fact that information is not there the fact that we have information out in the realms of internet that has biases towards particular demographics and we've seen with early versions of LLMs how they've maybe spit out particular assumptions such as occupations based on gender and other socio-economic characteristics. And if you're building something for an external facing company, you don't want to be the company that's in the news. And we've seen countless ones of these like DHL last year and others that end up in a situation where they say things they shouldn't. So we need to be careful about that too. We also need to think about language ambiguity, not necessarily in talking about the idea of dealing with multiple language, but the fact that even us as potentially native English speakers or even in your own native tongue as well. we can have situations where you know different words can mean different things and as someone with six year old who's having to teach them you know reading and they're going through the school system i see this pop up all the time and it's something that can pop up with these models as well and the last thing is a really wacky one called catastrophic forgetting has anyone heard of this one one person so it's kind of like what it sounds it basically you know through maybe poisoning or some other particular kind of operation it ends up forgetting massive chunks of the data it was trained on it's not fantastic but all of these things can happen but one of the other things aside from these factors that we need to think about for the reason for hallucination is quite frankly, they have been incentivized to through the training and evaluation procedures that they've been undergoing. So this is a paper that came out in September of last year, and it's from researchers at OpenAI. And what they're putting forward is the case that the training and evaluation procedures that these models are going under are rewarding guessing over acknowledging uncertainty. So everyone's had an example as we saw with the kind of quick pull of hands where it said something wrong they've been incentivized to do this to give us an answer at all costs and therefore we shouldn't be surprised that sometimes they're going to come out with information that's not quite there anyone who's got children will know that if you or even pets if you think about kind of training if you're incentivizing that particular creature to do something specific and you put in the wrong incentive they're going to do the wrong thing and these models are very much in that regard the same. So we need to provide information to make sure that they actually do what we the task and achieve the tasks that we want them to do. And for that we have several different tools within our toolbox and these together make the field of context engineering. and we're going to talk about each of these in turn with an example so by the end you'll know about not just what these things are within a system but also the techniques you can leverage and also resources to go off and learn more about it and the example we're going to use is this one so this is my little travel chat bot and his name is sorely And what it does basically is it takes results from an LLM based on the user provided input of where they want to go. It generates an itinerary and also leverages tools to pull in information from other various sources. It's written if anyone's curious within using TypeScript, AISDK, if anyone wants to dive more into that later, come and find me. So let's dive into context engineering and talk about these principles. So the first one is the user prompt and we all know what this is because we're entering them into LLMs every day. It's the input that a user is going to be putting into the large language model. It might be text, it might be something else, but this is the initial framing. Do we have a lot of control over that? Well, certainly there's things we can do with regards to UX design and kind of prompting and placeholders to try and make sure that we kind of put forward exactly what they should be doing. So in this particular example, it was very generic. It's what you want to ask because it's Google Gemini. And it's been able to get me a rather long answer about recommendations for which car to buy. But it's not just that as well. We can add in now with the kind of inclusion of multimodal models that can handle images, video and other forms of medium. We can provide prompts that have images capabilities as well. So this is an example here where I've said I have this diagram. I want you to basically change the background to black and white so it looks better on my slides. And as you can see, it's gone and managed to do it for me. So. How does the prompting look within our agent and how do we make sure that that gets back to the right place for the LLM to take action on it? That's what we're going to dive into now. So here I've got my little example travel planner. So let's say I want to go to Berlin to see the sites. Now that's going to... Demo gods, what are you doing to me today people? Is this because potentially... Ah, it helps if you start the app, doesn't it? Otherwise, you know, what's going to go wrong? NPM run dev. There we go. So if I copy this and do this again, hopefully we shouldn't get the same problem. Hey, there we go. It helps to start it, doesn't it? Right, okay. So how is that input getting back to the LLM is what you're going to be asking. Well, I'm using AISDK, there's numerous different frameworks out there, but the principles are very much the same. What we're going to have is we're going to have a scroll all the way down to the bottom. This is a big page. You see we've got an input element, okay? And what it's doing is this is our search box that we're going to get the prompt from. There's no complex magic in this particular web tool here. We've obviously placeholder with where you would like to go, which is our prompt to the user to try and limit what they're doing. But this doesn't replace mechanisms such as prompt sanitation, checking, using guardrails to establish whether the prompt is within the category of things you want to deal with, that they've not submitted for something sensitive or illegal that you shouldn't let them do. You need to use classification tools to sanitise prompt and make sure that's not happening. That's not. that's not in this particular example and then you'll see that when i hit the enter key it's going to send the user chat and the way it does it here is because this is using a react hook called use chat all it's going to do is basically send the message and we need to make sure that it's in the respective contract that's expected because it's not just the text and that's it that's it LLMs tend to want to have the full stream of messages including things like the role so is it from the user is it one that was generated by myself as the system what kind of parts is it type text is it images what's the format that I need to process here and then of course the actual message so we are going to send that through and when we do send it through it's going to hit this end point here which is called slash API slash chat in this instance. And you'll see here that we're pulling back the respective messages from the JSON request. And then we will pass them in via the messages here, utilizing stream text. And what you'll see with a lot of frameworks is you've got the option to generate text, object, whatever it is, or you've got the option to stream. Streaming, in my view, is better for the user responsiveness of these tools because it means that you're getting back the response in chunks and are able to show a gradual progression of what's going on. Meanwhile, if it's a text response coming back via generate, obviously they're going to have to wait and then you are going to have an endless spinner going on until the end of eternity and means most of the time they're going to log off and go somewhere else. So you need to think about the user experience with regards to interaction with the prompt as well. and that means you end up with something really simple with this so you can see that it's adding in relevant information including the type travel itinerary and some flight information which is all generating itself okay so moving on we need to talk about instructions. Now instructions in terms of system prompt tend to be used interchangeably but in simple terms these are the set of instructions that we wrap around the messages to say this is what you need to do. We're outlining what the task is, what things the LLM should be able to do and what it's not able to do. It's not just a case of spitting out you know a simple question and it's that there's a whole field on designing prompts called prompt engineering. So who's heard of prompt engineering? probably a big chunk. Yeah. So prompt engineering, we actually consider that to be a subset of contract context engineering, because it's basically responsible for one of the elements within the context engineering toolbox. So we're not saying prompt engineering has disappeared that it's dead. It's simply a part of context engineering. When it comes to prompts, there's numerous different elements that we can tweak and control in order to try and make sure that we get the right result and provide the instructions because like my maths teacher used to say in high school, rubbish in, rubbish out. That is not just applying to the context we apply to these models. It also comes down to the prompt and the instructions as well. If we give contradictory or incorrect information in the instructions, it's going to bamboozle it just as much as passing in contradictory documents is going to. So the elements that we have within our prompt as you can see up here is firstly you've got the notion of the role and the role is the persona that you want the LLM to adopt and this is something we do need to think about not just in terms of what we provide but what potentially an attacker could provide to try and do nefarious operations. So here I've got the persona of a helpful travel agent which seems quite innocent it obviously aligns to the thing I wanted to do which is to generate travel itineraries. But what if an attacker comes along and says, oh, I'm going to change this to a pirate? It's actually quite a common attack technique. And if you've not sanitised your prompt correctly, there's a potential that it might take that persona and adopt, you know, a slightly immoral character, let's say, and therefore be more likely to perform actions that you don't want it to do. So maybe extracting confidential information or saying things that you don't want it to say that might cause reputational damage. So we need to think about the role carefully and also whether there's a potential for it to be overridden. The directive is the next of the directive, much like prime directive, is basically the instruction. What do you actually want it to do in simple terms? We've got the output formatting, which is basically what format do you want it to be in? Now most people think, oh, well, that's going to be text. Actually, LLMs can generate objects and structured data, which we'll talk about later. You've also got examples. Now, examples can be really useful for trying to help it understand what it needs to do and also be able to apply that knowledge in order to be able to perform the action successfully. So I've got an example here saying, you know, if it's somewhere like Glasgow, it's going to be raining. If this was Aberdeen, obviously, I'd probably say the same thing. But you get the idea in that instance in saying recommend indoor activities because if you're, you know, going around in the rain to see a city you probably want to find something inside and keep dry we've also got style instructions which is exactly what it says on the tin it's basically you saying how i want the data to look when it's output in that particular textual format in this case and then finally we've also got additional information which can be useful for things such as guard rails or kind of anti-pattern conditions that you want to put in so basically saying don't do this. So I've got an example up here where it's saying if the result from the common commonwealth and development office tool that I have says don't travel to this area don't generate an itinerary and let them know why. So this would be something obviously that you could put in particular guardrails and say don't do this and then hopefully it won't do it. Now, it's not just the notion of these elements within a prompt. There's actually numerous different prompting strategies, many of which are summarised in the Survey of Prompt Engineering Techniques paper, which I've referenced at the top. And all of these are basically different types of prompts, different techniques of structuring them in order to tailor what you want an LLM to do based on particular patterns or even the type of problem that you're trying to solve. I'm not going to cover every single one because that's quite a weighty paper. I'd recommend going and checking it out. But what we'll do is we'll talk through a few key examples that you might want to leverage in your own systems. So the first one is we talk about zero versus few shot prompts. A shot prompt in simple terms, it's got examples in it. Those exemplars that we talked about before. So if it's zero shot prompt, I'm going to poll the audience for a question if you want to shout out. How many examples are we thinking a zero shot prompt is going to have? Zero. Yes. It's really simple. It's none. Okay. It's just going to be the instruction and that's it. Meanwhile, I've got another example here of a three-shot prompt. So I guess pop quiz, how many examples are in that one? Three. See, we are awake. It's all good. So the next one is better for reasoning tasks and it's a type of prompt called chain of thought and actually there's new if you look into the research there's numerous different types of alternatives and kind of deviations of chain of thought prompting that have appeared but the general idea of chain of thought prompting is that basically within your input to the model you're putting in a particular example along with the reasoning as to how to calculate the answer and for that reason these are quite good at reasoning and logical problems so as you can see from the example i've got up there at the top you'll see that compared to this on the left where i just say you know the answer is 11 and that's it we've actually outlined the logic as to why the answer is 11. And then what's happened is with the next question that it's answering, it's actually went and used similar logic in order to get the correct answer. So if you are utilising reasoning tasks, it's worth trying to structure your prompts in this particular way and provide logic examples to help it understand, because otherwise it might get the wrong answer. And that can be really embarrassing. But as I said, there are ways to extend that. So one valid extension of chain of thought prompting is called self-consistency. now chain of thought depending on the number of examples that can get quite weighty and can get quite complicated so the idea of self-consistency is it basically samples the different paths put into the few shot chain of thought prompt and then what it's doing is it looks for the most consistent answers coming out through from those paths and it uses that to determine the right answer i've also got meta promptings anyone heard of meta prompting before yeah has anyone done any meta prompting Maybe. So what meta-prompting is, you can see the example spinning up there, is me putting my prompt in and saying, can you make my prompt better? This is important, not just because it might find us ways to optimise in terms of taking in lengthier prompts and reducing it down, which helps us reducing the number of tokens that we're sending to a large language model. It can also point out inconsistencies. It can point out situations where maybe the order of things could cause issues with an LLM. And you can also get it to basically generate particular enhancements based on the LLM you're going to use so that you can try and leverage the best practice techniques for that model. because believe it or not it's not as simple as swapping one model in so background to this two weeks ago i thought you know what i saw some models that came out over christmas i'm going to have a play and one of them i decided to have a play with was function gemma because when i'd used gemma before it didn't have tool calling support so that kind of scuppered me with using it um so function gemma just literally switching out the model didn't go so well. So instead of generating an itinerary, I basically got a message saying, I can't do this, which is useless. So it kind of puts forward the notion and importance of traditional techniques still apply to genetic systems, like testing, like validation, like not swapping the things immediately out and expecting the exact same thing is going to happen. Even swapping a minor upgrade to a large language model or a major upgrade version, think about switching from version 4 to 5 of ChatGPT, can have the situation where you don't end up with the right results. So it's actually important to make sure you have, you know, a set of sample user inputs and validation to make sure that the model is going to work before it hits production. So a couple more prompting techniques before we move on. So this is one called REACT, not the framework. And it basically stands for Reasoning and Act. And it's a pattern basically where you prompt the model to say, I want you to explain what you are doing and output the elements in this kind of... format here of you know what's your thought what are you thinking at that moment time what's the corresponding action that you think off the back of this and what's the observation that you see and as it goes through a few times you kind of by spitting out this logic hopefully it should kind of one help explain what's happening and two should lead it to the right answer And then just like we do in Linux and operating systems, we can chain prompts together. It doesn't have to be a one and done situation. So this example here at the top, you can see that I've got one particular prompt where I've got this result that spat out here. And then with the second prompt, I've got in a new piece of information. And then we've also added in the result from the first one. And so just like chaining pipes within the Linux environment, we're all familiar with that. So when it comes to system prompts within this particular tool, what we need to do is pass in the system prompt as part of stream text. And that's going to be combined with the messages as our total context window for this particular agent in order for it to understand what is going on. So what you'll see is I've got a prompt here. it doesn't have every element that we talked about with the structure of a prompt you can see it's got helpful assistant oh i don't want the definition that's fine helpful assistant is the persona the directive is it's going to return travel itineraries and then we've also got an additional information here saying don't i generate an itinerary does it have everything it doesn't say the format so obviously the lm will therefore assume text But past then alongside the messages, that's basically how we ended up with this particular result here that we showcased before. So in terms of how to build a good prompt, you need to think about several things. You need to think about the task type because the task type is going to help you understand what particular prompting strategy is going to help you. So do we need to consider a chain of thought prompt versus, you know, something of a more traditional structure or maybe something else? You need to think about the complexity and the ambiguity both within the task and within the prompt that you are using. If you end up having a really complex prompt with perhaps too many examples similar to the construct of overfitting, your LLM doesn't have any flexibility in where to go. That might be what you're going for, but sometimes this can actually impede the LLM's ability to generate the desired result. Conversely, with ambiguity, if you don't give enough examples, you shouldn't be surprised if it ends up spitting out some wacky results. We need to also think about inputs and outputs. Again, math teacher, rubbish in, rubbish out. Applies to software systems, applies to data transformations, also applies to large language models. Think about the format of the data, both that's coming in and coming out. LLMs are good at reading particular types of structured data, but they might not be as great for others, so you need to think about that too. Think about the persona to emulate. I give one suggestion, maybe don't use a pirate. And then of course, think about the LLM itself. You know, there's particular shorthands that some LLMs are actually good at spotting that are unique to them. So you can always leverage them within your persona, within your prompts that you build. So the next thing we need to talk about is RAG. Now, I think I know how this is going to go. How many people have heard of RAG? And I'm not meaning red, amber, green statuses, by the way. Okay. So most of us have, right? We're going to go through this a bit quicker then because then we can focus on the bits that people don't know. So RAG, for anyone who didn't put their hand up, stands for Retrieval Augmented Generation. There have been several talks on the topic at this conference and others. I wouldn't be surprised if there's been workshops as well. And the idea is that we basically take out relevant documents or information that serves as the context to the LLM alongside the prompt so we can pass it within the prompt. Traditionally, we think about using vector search in order to find semantically relevant documents. Working at Elastic, I would argue, hybrid and lexical, depending on what you're pulling out, seem reasonable to me. It's retrieval, it's retrieving documents at the end of the day. But what tends to happen is you pre-process the documents. One, potentially by breaking larger documents into smaller chunks, which is the process known as chunking. for text documents obviously if you've got huge massive research papers getting the semantic similarity of the entire document may be different to the semantic context of one particular chunk of it so it's actually a good way to kind of help with with relevance and accuracy you then generate an embedding basically use a different type of machine learning model known as an embedding model to generate a massive stream of numbers and then we store those in some form of vector database so then when the user's inputting a query we encode the query using the model in the same way use that to find the semantically relevant documents and we'll talk about how they do that in a minute and then from there basically the documents get passed alongside the system prompt and everything else through the LLM to give us the final response So embeddings in terms of the numbers. Yeah, do you have a question? Of course. It's adding more information, yeah, but it's pulling it from somewhere. Yeah, that is. Oh, yeah, you're getting ahead of me. But yes, that is one way you could do it. We will talk about that when we get to tools later, but you can pass it in with the prompt as well. It just depends whether you've got tool calling support, which would be another thing to think about. Does that make sense? Cool. Right. So. Back to embeddings. Basically, the way that embeddings work is each of those numbers within that scary array that you've seen within SPIT as output to embedding models represents the intensity to which that particular document meets that criteria of a data aspect. And these data aspects are basically features that have been identified through the training process for that particular model. So when we have that particular situation, you see that at the intensity they'll have kind of a stronger value for that particular dimension compared to others. So if you look at this example here, we've got the notion of realism as one dimension, if you will, and then humanity being the other. So if I take this cartoon Vader as an example, you know, obviously he's cartoon, so he's not going to be on the realistic side, so he's going to lean towards the cartoon edge of the dimension. so we'll have a stronger intensity there but given he's also more machine now than man because of all the parts that have been added on he's leaning forward away from kind of human into more of the machine side so that obviously impacts where they are in terms of positioning but as you add in more documents what you find is that similar documents when the space starts to naturally group together anyone that caught the session on embeddings yesterday would have seen the same thing with the dog pictures all gripping together so you can see the idea that we've got Luke, Lea and Hansel up at the top there because they're all kind of realistic humans they've grouped within the same construct within our vector database so then when we actually do the embedding and generate for a sample query we then end up doing similarity and there are numerous different similarity algorithms out there. In plain English the idea is that you're going to take the query vector and the document vectors and compare the angles to see which ones are closest together and as the similarity tends towards one then it's considered to be semantically similar and therefore it will be scored more highly towards the top. So a cosine similarity is quite a traditional measure used by several different vector databases including Lucene, Elasticsearch and others and you'll see that the cosine of zero tends to one for as the angle tends to zero. Conversely, it tends to zero when they're orthogonal, i.e. at 90 degrees, for anyone panicking about the high school maths reminder. And then finally, for anything completely opposite, on total opposite sides of the vector space, it's going to be tending towards 9 as 1. Now, cosine similarity is not the only measure available. One of the limitations of it is that if you have loads of vectors with different magnitudes, i.e. they're different lengths, that therefore means they're still going to resolve to the same angle. So in that instance you probably prefer dot product, which is an option that you can often kind of change the algorithm for different vector databases. And then there's also one called Euclidean distance, which actually works surprisingly well for bit vectors. So there you go. Now, those comparisons are basically happening through an algorithm called k-nearest neighbor. It's performing those comparisons as we showed before. There's numerous different types and variations of nearest neighbor algorithms. This example here is HNSW, which is hierarchical navigable small worlds. And the idea is with HNSW is it splits the problem into layers, like many computer science constructs, splitting it into layers to try and manage the problem more effectively. So when we're searching for vectors, you can see from this example here, we specify the field where the vector is. you know, the vectors that have been generated. You're going to specify k, which is your number of candidates. One of the things that's specific here to Elasticsearch is number of candidates. So that's the number of documents you will compare in each shard against the query vector that's come in. And then, of course, you do need the query vector, otherwise you're not comparing anything. Now, the thing with vector databases and vector comparisons is if you have a lot of vectors, you're going to end up with the situation because it's all these mathematical comparisons. If you have millions of vectors, you have the potential to say that your query is going to take quite a long time to respond. And we don't want that. Sometimes we need to look for optimizations. One technique that you can utilise is one known as pre-filtering. And the idea here is that you actually use traditional lexical to basically narrow down the number of eligible vectors that therefore need to be compared against the query vector. And we shouldn't be afraid saying that, you know, lexical is not useful. It can be combined in all sorts of different ways. And this is a good example that will help you narrow down the vector space. So, for example, if you don't want to compare every single document and you're focusing on a particular time frame. There's no reason that we can't use temporality and look at the timestamp and say, well, actually, I only want to look at vectors from the past week or so. And we shouldn't be afraid of that. But even then, we just talk about retrieval and that's it. Sometimes the results might not be where you expect it. And there's actually another classification of models called re-ranking models. So who's heard of re-ranking models? A lot less. Cool. So the name on the tin basically gives you the clue as to what this is. It is a model that will take in the set of documents. It will basically generate a new score for each document based on the features of interest that are kind of embedded within its parameter set and it will return a set of documents with new scores that have been re-ranked. There's loads of re-ranking models out there. So there's some common ones from Quen, there's Gina, there's one called a where was it mixed bread which i found quite an interesting name haven't dived into it um and all of these are publicly available so you can actually you know leverage them from hugging face and other public repositories and give them a try if you want to see if they work for you so let's get on to that rag example shall we so i don't know if anybody noticed but you'll see here that we've got flight in here now it's not made this up we've provided this as context by amending our prompt in this regard. You can do it with tools as we found out from that great question, but we'll get to tools later. So what you can see from this prompt is that before we go ahead and define the prompt, I've got this utility called GetFlights. And all GetFlights is doing is it's actually doing a multi-search. So it's actually initiating two queries at once. Because, you know, I don't know about you, like, I love traveling and going on holiday, but you do have to come back to reality at some point. So you do need a return flight. Can't just kind of go out there and leave all the worries behind. So what you'll see is in the first query, we've got, you know, that the matches for origin and destination. We could use semantic here, but given we're probably looking for precise destinations, it's probably best to use lexical in this regard. And this is what I was talking about before of actually think about which mechanism makes sense based on your data. And then you'll see the return leg. We've basically flipped it. So you're going from the destination and coming back to the origin. This is really simple. Doesn't assume multi trip. I could make this super complicated, but we've left this as easy as we possibly can. So once we've got those respective results, If we actually get relevant results back, we also say I want you to include the following flight options and pass those in. So then that is all going together via the system prompt. Cool. So let's move on. So we need to talk about memory now. And we're going to cover short and long term together because they are related constructs. And what I mean by memory here is basically ways of storing the messages and the context from what's been happening with the conversation as we've been going or what's been happening with the actions. Much like humans, I'm not from a psychology background, but much like humans, we have the notion of short and long term memory. We have notion of short term and long term memory for models as well. Short term memory is the current session. So thinking about what we showed before with the prior example, that is one session. So if I ended up adding more messages, started interacting with it more, you would actually see that we'd end up with something similar to this with several different messages. And it's also the notion of the human and the AI messages as we talked about before, it's going to keep all of those within that current session within memory, and that's called your short term memory. Long term memory is memory leveraging some form of store in order to try and store messages that might be relevant later on. So those might be, you know, knowledge of prior sessions, things you've asked about before. Maybe we want to pull out itineraries that have been proposed to you in the past because, you know, we think you'd be more likely to book it now. Maybe you've had time to go and have a think about it. Or alternatively, do we want to expand greater than that? Have there been itineraries generated by other users that might actually match your preferences and we can recommend those instead? This can be a good way to try and ground and limit the context as well. In terms of where we can store these memories for long-term memory, there's several options. You can use a file system, you can utilise a database, you can utilise a vector database if you want to leverage semantic retrieval to pull the memories back. There's all sorts of different options. It partially depends on what you're wanting to do and how... you want to manage those memories as well because llms can handle semantic as we've seen with rag but they actually are surprisingly good as anthropic found at traversing through file systems so that is a valid approach for saving them in files as well so we are going to move on to another demo but firstly i need to change the branch so if you just bear with me run start yeah there we go cool okay So here we're going to ask another example just to take it through and show the next set of results. So this time we'll go to Amsterdam, shall we? So while that's spinning through, let's look at the code here. So what you will see in the code that has been changed on the root is I have this additional message here. So you've got the messages coming in here from the current session. So that's the short term memory. And these need to be converted into model messages, which is a quirk of AISDK because it differentiates between those. It differentiates between the system messages and the human messages that we've seen. But if I want to get a hold of similar messages, what I can do basically is get a hold of the location and then I can search matching the message content. And given this is actually a semantic field, this is a semantic match. Sorting by timestamp. I've specified the size as 20. You also need to think about this context overflow element again. Maybe returning 20 is too many itineraries and maybe you should consider returning only one or two. So that's another thing you can think about with regards to limiting of the context window as well. And then we're going to return the particular hits here. Then what happens next is you'll see... when we actually run through it's got all of the messages so it's been combined with the short-term and long-term memory messages and indeed we're going to persist the messages at the end so we're persisting the itineraries so that they can be reused at a later date and we're persisting those back to our index and as you can see it then is giving us outbound and inbound flights as well alongside the other things it's also picking up kind of the same kind of itineraries based on what's happened before so you'll see here we've got rick's museum so if i go in here and if i do a simple match there we go nothing like some live coding it's title no it's not no it's not flight data it's the other index that'll be why i can't find it message.context there we go so if i have a look for this you'll see actually we've got 12 so it has been persisting the same thing and returning the same thing time and time again. Now when it comes to memory as we build out more complex agentic systems you get into the notion of sub-agents which is basically you know agents depending on other agents we can have the situation where providing memories and context actually really depends on the individual agent because For example, if agent one has relevant output for agent three and we don't pass that, that can impact the result of the entire chain of agents. And that can lead to us not getting the desired result. So in addition to thinking about the overflow of individual sub-agents, we also need to make sure that the relevant memories and context is being passed between them. But we also need to manage it. It's not like the age-old days of log files where you get a 95% warning that the file system is about to basically run out of memory and you frantically try and delete old log files manually because you haven't set up an archive task yet. We still need to think about how to manage these and there's two particular techniques that we can think about. So the first is we can use summarization of the messages in order to make them shorter. There are several Reranker models available that can do this and one example is the province model you've got here that you can see the paper. And what you see from this example is it's actually looking at the question and figuring out which particular sentences are not relevant. You can also use LLMs for summarization, that's another valid approach, but you've got both of those as an option there. And then we've also got the notion of pruning based on particular data aspects as well. So if we dive into our demo again and just go straight to the code this time, the first thing we're going to go is back to that break point that we had before where we talked about summarization. So what's going to happen in here is that I'm asking a different model to generate. a summarized message so what you'll have seen is in my original route there we go i am connecting to gpt4 using azure and when it comes to the context i'm actually utilizing a locally running model version of quen 3 and i'm utilizing a tool called olama so it'll take a wee moment to come back I've passed in a prompt saying I want you to summarize the following message and then of course I've included the message because asking it to summarize something and not having it there as we all know from context by now we're going to end up with a result that you know is not expected and has a particular problem. So that's going to take a particular moment to come back. There we go, okay. Clearly I've put my laptop under pressure this morning because a llama is taking a while. There we go. But what you can see is if we go and output the counts, it's taken the original message of 27,024 and then it summarised it down to 13 and 12, which it then returns and is going to persist down into our index. But we also need to think about over time the messages accruing within that index and getting rid of them. So we also need to think about the idea of pruning. And what pruning is, is the idea of deleting irrelevant messages because remember what we talked about with regards to the idea of messages that could potentially poison the context if you have out of date information irrelevant information you don't want to be pulling these back so pruning to remove irrelevant or outdated information is vital so here you'll see that We've got, we're going to be pruning the messages. So the initial count is there's 30 sitting within my index at the moment. I'm going to engage in what's called temporal pruning, which is basically I'm going to delete anything that's older than two weeks. So we'll do that now. And then hopefully if everything all goes okay, it says it's pruned 15 messages from the index. So if I now do it again, it's now saying that I've got 15 left. So it's actually reduced it by half, which is quite good. Cool. So we'll talk about this one in a minute, but I'm going to get that running while we go and talk about our next topic. So the final thing, no, second final thing, penultimate thing that we're going to talk about is tools. And this relates to the question that we talked about earlier with regards to passing in relevant queries from that flight tool. So we'll get to see what that does. Tools, they're functions, basically. They're functions. That's why we call it tool calling or function calling interchangeably. It's the idea of we have a particular function that we've defined and we've said to an LLM, if you need to do this, this is where to get the data from. This is how we can not just get them to return data, but return data from relevant sources to us and also potentially how we can get them to perform actions. They come within two particular main fields, both in terms of agents that have defined tools that can communicate with each other using what's called A to A. A to A is agent to agent protocol. devised by and proposed by Google and it's basically a contract much like the protocols we use within you know regular everyday building the software where we end up basically establishing a communication conduit between agents so the way that it works is you have the notion of a model card and not a model card sorry an agent card and what happens with an agent card is it defines details of what the agent does it tells you what tools are available and the agent that's basically initiated the request can then say well i think therefore it can help me with this particular task which is part of my problem space so it then can initiate requests to have the other dependent agent pull in the relevant results invoke tools that it needs and then generate a response You then also have the notion of model context protocol and certainly there's dedicated talks and workshops that have been at the conference talking about model context protocol in a lot of depth. But what you need to know in simple terms with regards to tools in this regard is that you have a server. model protocol model context protocol server that's able to basically have source agents communicate with it and that server can expose various resources that can be used by that agent so key things would be the tools that we're talking about invoking apis pulling out relevant data, performing particular actions. Obviously you need to think about permissions and things like that when we talk about actions and don't do anything too destructive. But also you can utilise it for pulling out resources and you can also utilise it for having saved kind of regular prompts as well. But not every single model supports tool calling, so we need to be careful with that. and this is an example if you're ever looking to try and find how effective a particular model is with regards to tool calling you should check out the berkeley function calling leaderboard and it gives you ideas of basically the top models from their evaluation is updated on a regular basis and it also covers particular things such as is it good at web searching is it good at multi-turn all these other kind of things so if we go back to our demo to go through the tools What you'll have seen already from one of the prior runs is that we have three tools here. We've got one that's pulling in the flights, one that's pulling in the Foreign and Commonwealth Office data, and one that's pulling in the weather. And all of these aspects are therefore then used to generate our itinerary. So what you will see within here is going to the route, I've defined three tools at the top, flight, weather and FCDO. If I deviate and show you the Flights tool by comparison to what we saw with the RAG example before, you can see within my tool I have particular attributes that I've defined to help explain to the LLM what this tool does. So I've provided a description, which is really important to tell it what it actually does. I've provided an input schema to tell it the contract that's needed for the particular function, and this obviously helps us enforce type safety as well. and this is using a type schema library for typescript called zod has anyone heard of zod a few yeah so very common but integrates really well with aisdk and you'll see specified the destination and origin and strings with description and then of course i've defined an output schema as well these are optional but i find them very useful and you'll see that i've specified an object and i've actually used the custom from zod to pull it out of a flight type and then describes it as basically the output and return flights. You can mark particular attributes as optional as well, but here obviously everything is mandatory. And then we need to specify our function using the async execute option, passing in the relevant parameters, and you'll see it's doing the same M search that we were doing within the RAG example before. But this also gives us additional power when it comes to building out user experiences, if we think about it, because we can then actually utilise the output of a tool in order to showcase the relevant results and generate more meaningful, relevant experiences for them that are custom. Because obviously, if there's particular tools that are not relevant to their flow, we can choose to not show those, or we can choose to show the ones that are indeed relevant and give a more personalised experience. So if we go down here and I'm going to find my, there's a lot of tools in here, tool flights. You'll see when the input's available, I've got a placeholder for loading because we've not received our output yet. But then when you see when the output's available, it's utilising a flight tool, which just passes in the respective props to generate the results. And that's how we end up with this particular tool result over here. Now, when it comes to tool calling, Sometimes it can be difficult to figure out what the LLM is doing under the covers because at the end of the day the LLM is making a decision partially based on its training and its own kind of system limitations with regards to how to what degree it supports tool calling and multi-tool calling. So for example I've had particular models sometimes that will only call one tool and they will pick the most appropriate one and that is it. So if I want it to return three tool results that's not very helpful for me. But it also shows the importance of observability and being able to actually see what's going on so this is an example of a trace i've taken for this agent using open lit and you can see which of which there was a session yesterday on if anyone was curious and what you can see within here is within it i can see the traces for the start text run and i can see in here every single tool call which ones are what order they're being called in how long they're taking and indeed what dependent systems they're invoking as well and this is going to prove to be really important in actually trying to diagnose potential issues in production and also identify changes in behaviour. Is it suddenly changing the number of tools it's invoking or is it changing the order? Because that can obviously impact the final results as well. We also need to think about should tools even be accessible at all to this particular model? and that obviously falls on from the principle of least privilege if we think about security. You know, maybe it shouldn't have access to every single tool that you've defined. So this is an example here, it's actually from Elastic Agent Builder and you can see that this example is only provision 5 of a possible 24. Partially to limit, you know, what the responsibilities of that agent are. but also partially to prevent bamboozling because the more tools that you provide to an LLM the more complex the problem space is and in fact sometimes it can even have difficulty choosing which are the most appropriate tools so you need to think about exactly which tools are being passed in and not just say oh I can use tools so I'm going to throw the kitchen sink at it and have 50 100 tools in that instance you need to start thinking about how to optimize the tool capture and there's various different ways you can do that but one promising one based on this paper from 2025 is one called RAG MCP. And what this is is basically using semantic text on the descriptions of the tool to pull back which ones are considered relevant for that task and passing that as part of the context of the LLM in order to basically improve the accuracy. And as you'll see from the result up there, what they're saying is that they cut the prompt tokens by over 50% and they tripled the tool selection based on their experiments. So this actually shows some quite promising results. And the last thing we need to talk about is structured outputs. And structured output should not be a surprise to anyone, you know, anyone who's familiar with things like open API specifications, schemas, anything like that. The idea of being able to specify the structure and contract of our data is also relevant for LLMs as well. And we need to remember that. So the idea is that you specify a particular schema down here, which then means when it goes to generate the final result, it's going to generate it based on that schema. so we can enforce that particular contract going to our final demo this one's a bit different so i invoked this earlier while we were talking so i'm going to walk through it hopefully people can see this okay so what i have here is a color request going to a new endpoint which is called api slash itinerary and we're passing it in an origin and a destination that particular root is here So if we scooch this down a second, you will see that what happens is we're pulling these out of the request. We're using a generate, which is why I initiated it earlier, because it was going to take a minute, utilizing a local model and specifying a prompt that we want to generate a detailed itinerary for a trip from origin's destination and say, I want you to include the key sites, things to do, hotel and dining recommendations and suitable flights. But then we specify a schema. So here is the output object that we want it to spit back, so we want it to have origin and destination, we want an array of flights, we want the hotel, and we also want that day-by-day itinerary that we've been generating through our other examples. And as you can see from the terminal output, good old-fashioned blue Peter style, here's one I did earlier. What you can see, if I zoom in a bit more, there we go. Here is our result, so it spat it out as text. but you'll see that we've got the relevant parameters so we can actually then process and convert it back into an object. So we've got the origin of London, the destination is Berlin. You'll see I've got the flights array in here. If you scroll down further, you can see I've specified the hotel, so it's basically figured this out from its own context. And then indeed you'll see that we've got a big long itinerary, similar to what we've been generating before. And this means that we can also, in addition to spitting out structured data that might be useful and easier for humans to understand, or using formats such as JSON, we can end up actually spitting out things that are relevant for input into other systems. So it's not a case of agents working in isolation anymore. We could potentially use them to pass in relevant information to more traditional systems that have an agreed contract. So we've talked about context engineering. We've talked about all of these practices and the ways that we can use them to manipulate the context. Because it's not just a case of relying on the training data. You might have data that's relevant to the situation. You might find that there's contradictory data out there that can lead to a poisoning of agents and problems. and certainly in the future i see that a big part of our role is going to be figuring out the right data to pass to these particular systems and we need to know therefore which tools we can leverage to do that including obviously actual tools that we've talked about here and also techniques such as structured output memory instructions user prompting and rag if you want to learn more a whole ton of resources that i found useful and a couple that myself and colleagues have written as well if you take a scan of the qr code that's where the code is as well as a list of these resources in the readme so don't panic and everything is there if you have any questions we've got a couple of minutes left if not come and find me at the booth and ndc it's a pleasure thank you very much

This talk was recorded at NDC London in London, England. #ndclondon  #ndcconferences #developer #softwaredeveloper    

Attend the next NDC conference near you: 
https://ndcconferences.com
https://ndclondon.com/

Subscribe to our YouTube channel and learn every day:   
/        @NDC 

Follow our Social Media!

https://www.facebook.com/ndcconferences
https://twitter.com/NDC_Conferences
https://www.instagram.com/ndc_conferences/

#ai #database 

We've all heard about prompt engineering. But now with the emergence of context engineering you may be scratching your head about what the difference is.

The reality is that to build reliable agents, we need to use all tools in our context toolbox to ensure they provide accurate responses in the correct format.

Let's dive into the world of context engineering, and understand the elements of context that we can tune and tweak to build more reliable AI agents and associated systems.