Context Engineering for AI Agents with LangChain and Manus

all right well thank you all for coming we'll go ahead and kick off the webinar now and i'm sure people will continue to stream in um i'm lance i'm on the founding years at lang chain and i'm joined by peak from manis um do you want to introduce yourself quickly yeah hey guys i'm the co-founder and chief scientist of manis so basically i designed the agent framework and a lot of things in manis and i'm super excited to be here today thanks for having me yeah We're really excited to do this because, first, Manus is a really cool product. I've been using it for a long time. But also, they put out a really nice blog post on context engineering a few months ago that influenced me a lot. So I want to give a quick overview of context engineering as I see it, and I'll reference their piece. And then Pete's actually going to give a presentation talking about some new ideas not covered in the piece. So if you've already read it, take a look at some things that are new. which hopefully will be quite interesting for you. But I'll set the stage and I'll hand it over to Peek and then we'll do some Q&A. So you might have heard this term, competition engineering. And it kind of emerged earlier this year. If you look through time with Google search trends, competition engineering was kind of initiated following ChatGPT. So that's showing December 2022. And when we got this new thing, a chat model, there became a great deal of interest in how we prompt these things. prompt engineering kind of emerges as a discipline for working with chat models and prompting them. Now, concept engineering emerged this year around May. We saw it really rising in Google Trends. And it corresponds a bit with this idea of the year of agents. And so why is that? One of the things that people have observed if you've been building agents is that context grows. And it grows in a very particular way. When you build an agent, what I mean is we have an LLM bound to some number of tools. That LLM can call tools autonomously in a loop. The challenge is, for every tool call, you get a tool observation back. And that's appended to this chat list. These messages grow over time. And so you can get this unbounded explosion messages as agents run. As an example, Manus talked about in their piece that typical tasks require around 50 tool calls. Anthropics mentioned similarly that production agents can engage in conversations spanning hundreds of turns. So the challenge is that agents, because they are increasingly long-running and autonomous, they utilize tools freely. You can accumulate a large amount of context through this accumulation of tool calls. And Chrome will put out a really nice report talking about context. The observation simply is that performance drops as context grows. So you have this paradox or this challenging situation. Agents utilize lots of context because of tool calling. But we know that performance drops as context grows. So this is a challenge that many of us have faced. And it kind of spearheaded this, or I think seeded this term of context engineering. Karpathy, of course, kind of coined it on Twitter earlier this year. And you can think about context engineering is the delicate art and science of filling the context window with just the right information needed for the next step. So trying to combat this context explosion that happens when you build agents and they call tools freely, all those tool messages accumulate in your messages queue, have to be kind of cull such that the right information is presented to the agent to make the correct next decision at all points in time. So to address this, There's a few common themes I want to highlight that we've seen across a number of different pieces of work, including Manus, which I'll mention here. Idea one is context offloading. So we've seen this trend over and over. The central idea is you don't need all context to live in this message's history of your agent. You can take information and offload it, send it somewhere else so it's outside the context window, but it can be retrieved, which we'll talk about later. So one of the most popular ideas here is just using a file system. Take the output of a tool message as an example, dump it to the file system, send back to your agent just some minimal piece of information necessary so they can reference the full context if it needs to, but that full payload, for example, web search result that's very token heavy, isn't spammed into your context window for perpetuity. So you've seen this across a number of different projects. Manif uses this. We have a project called Deep Agents that utilizes the file system. OpenDeep Research utilizes Actually, agent state has a similar role to an external file system. Cloud code, of course, uses this very extensively. Long-running agents utilize it very extensively. So this idea of offloading context to a file system is very common and popular across many different examples of production agents that we're seeing today. The second idea is reducing context. So offloading is very simply taking some piece of information like a tool message that's token heavy. and not sending it all back to your messages list, dumping it to a file system where it can be retrieved only as needed. That's offloading. Reducing the context is similar, but instead, you're just summarizing or compressing information. Summarizing tool call outputs is one intuitive way to do this. So we do this with OpenDeep Research as an example. Pruning tool calls or tool messages. One thing that's very interesting is, Plot 4.5 has actually added this to if you look at some of their most recent releases, they now support this out of the box. So this idea of pruning old tool calls with tool outputs or tool messages is something that Cloud has now built into their SDK. Summarize your compacting full message history. You see this with Cloud code in its compaction feature once you hit a certain percentage of your overall context window. Cognition also talks about the idea of summarizing your pruning at agent to agent handoffs. So this idea of reducing context is a very popular theme we see across a lot of different examples from cloud code to our OpenDeep research, cognition, Cloud 4.5 has incorporated this as well. Retrieving context. Now, this is one of the classic debates today that you might see raging on X or Twitter. The right approach for retrieving context. Lee Robinson from Cursor just had a very nice talk, and I'll make sure these slides are all shared so you can see these links. He had a very nice talk at OpenAI Demo Day. Talking about HERSR, for example, uses indexing and semantic search, as well as more simple file-based search tools like Glob and Grep. Cloud Coda Force only uses the file system and simple search tools, notably Glob and Grep. So there's different ways to retrieve context on demand for your agent. Indexing and semantic search, file system, and simple file search tools. Both can be highly effective. There's pros and cons we could talk about in the Q&A. But of course, context retrieval is central for building effective agents. Context isolation is the other major thing we've seen quite a bit of. In particular, splitting context across multi-agents. So what's the point here? Each sub-agent has its own context window, and sub-agents allow for separation of concerns. Manus Wide Agents talks about this. Our deep agents work uses this. OpenDeep Research uses it. Claude sub-agents are utilized in their research. Subagents are utilized in Claude's multiagent researcher and also Claude co-support subagents. So subagents are a very common way to perform context isolation we've seen across many different projects. Now, one thing I thought was very interesting is caching context. And Manis talks about this quite a bit. I'll let Pig speak to this a bit later, but I think it's a very interesting trick as well. I'll just show a brief example that we've seen across OpenDeep Research. This is a very popular repo that we have. It's basically an open source deep research implementation. It performs on par with some of the best implementations out there. You can check our repo. We have results from deep research beds showing that we're top 10. It has three phases, scoping of the research, the research phase itself using a multi-agent architecture, and then a final one-shot writing phase. We use offloading. We basically create a brief. to scope our research plan, we offload that. So we don't just save that in the context window because that context window is going to get peppered with other things. We offload it so it's saved independently. It can be accessed, in our case, from the line graph state. But it could also be from file system. It's the same idea. So you create a research plan. You offload it. It's always accessible. You go do a bunch of work. You can pull that back in on demand so you can put it at the end of your message list so it's accessible and readily available to your agent to form, for example, the writing phase. We use offloading, as you can see, to help steer the research and writing phases. We use reduction to summarize observations from token-heavy search tool calls. That's done inside research itself. And we use context isolation across sub-agents within research itself. And this is kind of a summary of a bunch of different of these various ideas across a bunch of different projects. And actually, Peek is going to speak to Manus in particular, and some of the lessons they've learned. This just sets up the stage. And this just summarizes what I talked about. These different themes of offloading, reducing context, retrieving context, isolating, caching, and a number of popular projects and where they're used. And a few different links. I will share these slides to the notes. And I do want to let Peek go ahead and present now, because I want to make sure we have plenty of time for him and for questions. But this just sets the stage. And Peek will let you take it from here. I'll stop sharing. Okay. Can you see my slides? Yeah. Okay. Perfect. Okay. Thank you, Lance. I'm super excited to be here today to share some fresh lessons on context engineering that we learned from building Manus. You know, here I say fresh lessons because I realized that the last blog post that you mentioned I wrote about context engineering was back in July. And yeah, it's the year of the agent. So July is basically the last entry. And of course, before this session, I went back and read it again. And luckily, I think like most of what I wrote in that blog still holds up today. But I just don't want to like waste everybody's time by like just repeating what's already inside that blog. So today, I think instead I want to dig into some areas that I either didn't go deep enough on before or didn't touch at all. So actually, we'll be focusing on the discourage column in Lance's earlier slides. Because, you know, personally, I think exploring those non-consensus ideas often leads to the biggest inspirations. Yeah. So here's the topic for today's talk. First, we'll cover a bit about the bigger question of why we need context engineering. And then we'll have more on context reduction, more on context isolation. And finally, some new stuff about context offloading, which we are testing internally here at Manus. Yeah, so everything I'm sharing today is in production in Manus. It's battle tested. But I don't know how long it will last because, you know, things are changing super fast. Okay, let's start with the first big question is, why do we even need context engineering? Especially, you know, when fine-tuning or post-training models has become, like, much more accessible today. Yeah, for example, folks at The Thinking Machine team, they just released the Tinker API, which I like a lot, beautiful design. But for me, the question like why context engineering actually came through several painful stages of realization. Before starting Manus, I've already spent over 10 years in natural language processing or NLP, which is basically what we call building language models, but before ChatGPT. uh menace is actually my second or third company at my previous startup we trained our own legacy model from scratch to do open domain information extraction and building knowledge graph and semantic search engines on top of them and it was painful our product's innovation speed was completely capped by the model's iteration speed you know even back then the the models were much smaller compared to today but still a single of training plus evaluation cycle could take maybe like one or two weeks and the worst part is that at that time we hadn't reached pmf yet and we're spending all that time like improving benchmark that might not even matter for the product so i think um instead of building specialized models too early uh startups really should lean on general models and context engineering for as long as possible well of course i guess now that's um some kind of common wisdom but as your product matures and open source base model gets stronger i know it's very tempting to think hey maybe i should like just pick a strong base model fine tune it with my data and make it really good at my use case you know we've tried that too and guess what it's another trap you know to make rl work really well you usually fix an action space design a reward around your current product behavior and generate tons of like on policy rollouts and feedback but you know this is also dangerous Because we're still in the early days of AI and agents, everything can shift under our feet overnight. For us, the classic example was the launch of MCP. Actually, it completely changed the design of Manus for make compact static action space to something like it's infinitely extensible. And if you have ever trained your own model, you know that this kind of open domain problem is super hard to optimize. Well, of course, you could like for massive effort. into post training that ensures generalization. But then aren't you basically trying to become an LN company yourself? Because you're basically rebuilding the same layer that they have already built. And that's a duplication of effort. So maybe after all that buildup, here's my point. Be firm about where you draw the line. Right now, context engineering is the clearest and most practical boundary between application and model. So trust your choice. All right. Enough philosophy and let's talk about some real tech. First topic, contact reduction. Here I want to clarify two different kinds of compaction operations because we think like constant reduction is fascinating, but it's also a new concept. There's a lot of way to do this. And here in Manus, we divide them into compaction and summarization. For compaction, in Manus, every tool call and tool result, we actually have two different formats, a full format and a compact one. The compact version strips out any information that can be reconstructed from the file system or external state. For example, here, let's say you have a tool that writes to a file, and it probably has two fields, a path and a content field. But once the tool returns, you can ensure that the file already exists in the environment. So in the compact format, we can safely drop the super long content field and just keep the path. And if your agent is smart enough, whenever it needs to read that file again, it can simply retrieve it via the path. So no information is truly lost. It's just externalized. We think this kind of reversibility is crucial because agents do chain predictions based on previous actions and observations. And you never know which past action will suddenly become super important 10 steps later. You cannot predict it. So this is a reversible reduction by using compaction. Of course, compaction only takes you so far. Eventually, your context will still grow and will hit the ceiling. And that's when we combine compaction with the more traditional summarization. But we do it very carefully. For example, here before summarizing, we might offload key parts of the context into files. And sometimes, we even do more aggressively. We can dump the entire pre-summary context as a text file or simply a log file. into the file system so that we can like always recover it later and like lance like just mentioned some people just use like globbing rep you know globbing rep also works for log files so if the model is smart enough it even knows how to retrieve those like summarized uh those pre-summarized context yeah so i think the difference here is that compaction is reversible but summarization isn't both reduce context lens but they behave very differently and to make the these uh to make both nether co-exist, we have to track some context length thresholds. At the top, you'll have your model's hard context limit. Say like 1 million tokens is pretty common today. But in reality, most models start degrading much earlier, typically maybe around 200K. And you'll begin to see what we call a context rot. It's like repetitions, slower inferences, degraded quality. So by doing a lot of evaluation, it's very important for you to identify that. pre-ROT threshold. It's typically 128K to 200K and use it as the trigger for context reduction. And whenever like your context size approaches it, you have to trigger a context reduction, but starting from compaction, not summarization. And compaction doesn't mean like compressing the entire history. You know, we might compact say like the oldest 50% of tool costs while keeping the newer ones in full detail. So the model still has Resh, you shot examples of how to use tools properly. Otherwise, in the worst case, the model will imitate the behavior and output those compact format with missing fields. That's totally wrong. After compaction, we have to check how much free context that we actually gain from this compaction operation. Sometimes in this graph, after multiple rounds of compaction, the gain is tiny because even it's compact, it still uses context. That's when we go for summarization. But also keep in mind that when summarizing, we always use the full version of the data, not the compact one. We still keep the last few tool calls and tool results in full detail, not summary because it can allow the model to know where left off. and will continue like like more smoothly otherwise you'll see like after summarization sometimes the model will change its style changes tone and we find out like keeping a few feel like total contour result examples really help okay now with that now we've covered reduction and let's talk about isolation i really agree with um cognition's blog where they warn against using multi-agent setups because like when you have multiple agents syncing information between them becomes a nightmare But this isn't a new problem. Multiprocess or multithread coordination has been a classic challenge in the early days of computer programming. And I think we can borrow some wisdoms here. I don't know how many Golang coders are here today, but in the Go programming language community, there's a famous quote from this gopher, do not communicate by sharing memory, instead share memory by communicating. Of course, this isn't directly about agent and it's sometimes even wrong for agents. But I think the important thing is it highlights two distinct patterns here, which is by communicating or by sharing memory. But if we translate the term memory here into context, we can see that parallel pretty clear. By communicating is like the easier one to understand because it is the classic sub-agent setup. Here, for example, the main agent writes a prompt and the prompt is sent to a sub-agent and the sub-agent's entire context only consists of that instruction. We think if a task has a short, clear instruction and only the final output matters, say, searching a code base for a specific snippet, then just use the communication pattern and keep it simple. Because the main agent doesn't care how the sub-agent find the code. It only needs the result. And this is what Cloud Code does typically using its task tool to delegate separated, clear tasks to some sub-agents. But for more complex scenarios, in contrast, by sharing memory means that the subagent can see the entire previous context. It means like all the tool usage history. But the subagent has its own system problem and its own action space. For example, like imagine a deep research scenario. The final report depends on a lot of intermediate searches and notes. And in that case, you should consider using the shared memory pattern. or in our language by sharing context. Because even you can save all that notes and searches into file and making the sub-agent to read everything again. But you're just wasting latency and context. And if you count the amount of token, maybe you're using even more token to do this. So we think for those scenarios that requires a full history, just use a shared memory pattern. But be aware that sharing context is kind of expensive. because each subagent has a larger input to pre-fill, which is like you'll spend more on input tokens. And since the system prompt and the access space differs, you cannot reuse the KVCash, so you have to pay the full price. And finally, let's talk a little bit about context offloading. When people say offload, they usually mean moving parts of the working context into external files. But as your system grows, especially if you decide to integrate mcp one day you realize that the tools themselves can also take up a lot of context and having too many tools in context leads to confusion we call it context confusion and the model might call like the wrong ones or even like non-existing ones so we have to find a way to to also offload the tools a common approach right now is like doing dynamic rag on tool descriptions For example, like loading tools on demand based on the current task or the current status. But that also causes two issues. First of all, like since tool definitions sit at the front of the context, yeah, your kidneys have resets every time. And most importantly, the model's past calls to remove tools are still in the context. So it might few shop the model into like calling invalid tools or using invalid parameters. So to address this, we're experimenting with a new layered action space in Manus. Well, essentially, we can manage to choose from three different levels of abstractions. Number one, function calling. Number two, sandbox utilities. And number three, packages and API. We'll go deeper into these three layers of layered action space. Let's start from level one, function calling. And this is a classic. Everyone knows it. It is schema safe thanks to constraint decoding, but we all know the downsides. For example, we mentioned like breaking the cache and maybe too many tool calls will cause some confusion. Too many tools may cause confusion. So in Manus, right now we only use a fixed number of atomic functions. For example, like reading and writing files, executing shell commands, searching files in internet, and maybe some like browser operations. We think these atomic functions have super clear boundaries and they can work together to compose like much more complex workflows. Then we offload everything else to the next layer, which is the sandbox utilities. As you know, each Manus session runs inside a full virtual machine sandbox. It's running on our own customized Linux system. That means Manus can use the shell commands to run pre-installed utilities that we developed for Manus. For example, we have some format converters, we have speech recognition utilities, and even a very special, we call it Manus MCPCLI. which is how we call MCP. We do not inject MCP tools to the function calling space. Instead, we do everything inside that sandbox through in the command line interface. Utilities are great because you can add new capabilities without touching the model's function calling space. It's just some commands pre-installed in your computer. If you're familiar with Linux, you always know how to find those new commands. You can even run like dash dash help. to figure out how to use a new tool. And another good thing is for larger outputs, they can just write to files or return the result in pages. And you can use all those Linux tools like grab, catch, less, more like to process that results on the fly. So the trade off here, like it's super good for large outputs, but it's also not that good for low latency back and forth interactions with the front end, because you always have to like to visualize. of the interactions of your agent and show it to the user. So this is pretty tricky here, but we think like it already offloads a lot of things. And then we have another layer, the final layer, we call it packages and APIs. You know, here Manus can write Python scripts to call pre-authorized API or custom packages. For example, like Manus might use a 3D designing library for modeling or call a financial API to fetch market data. And here actually, we've purchased all these API on behalf of our user and pay the money for them. It's included in the subscription. So basically, we have a lot of API keys pre-installed in Manus, and Manus can access these APIs using the keys. I think these are perfect for tasks that require lots of computation and memory, but do not need to push all that data into the model context. For example, imagine if you're analyzing a stock's entire year of price data. You don't feed the model all the numbers. Instead, you should let the script to compute it and only put the summary back into the context. Since code and APIs are super composable, you can actually chain a lot of things in one step. For example, in a typical API, you can do get city names, get city ID, get weather, all in one Python script. There's also a paper from one of my friends called CodeAct. A lot of people were discussing about it. I think it's the same idea because code is compulsible and it can do a lot of things in one step. But also, it's not schema safe. It's very, very hard to do a string decoding on Kodak. So we think you should find the right scenario for these features. For us, we use, as we mentioned, everything that can handle inside a like. like compiler and temperature runtime, we do that using code. Otherwise, we use like sandbox utilities or function calls. And the good thing is, if you have these three layers, from models point, all three levels still go through the standard function calls. So the interface stays simple, cache friendly and orthogonal across functions. Because, you know, we mentioned sandbox utilities, you're still accessing these tools using the shell tool, accessing these tools using the shell function. And also, if you're using APIs in third-party applications, you're just using the file function to write or read file and then execute it using the shell function. So we think it does not add overhead to the model. It's still all the things that models are trained and they're already familiar with. So let's zoom out and connect the five dimensions, offload, reduce, retrieve, isolate, and cache. You can find out that they are not independent. we can see that offload and retrieve enables more efficient reduction and stable retrieve makes isolation safe. But isolation, oh yeah, isolation also slows down context roles and reduces the frequency of reduction. However, more isolation and reduction also affects cache efficiency and the quality of output. So at the end of the day, I think context engineering is the science and art. that requires a perfect balance between multiple potentially conflicting objectives. It's really hard. All right, before we wrap up, I want to leave you with maybe one final thought. And it's kind of the opposite of everything I just said, which is please avoid context over engineering. Like looking back at the past six or seven months since Manus launched, actually the biggest leap we've ever seen. didn't came from adding more fancy context management layers or clever retrieval hacks. They all came from simplifying or from removing unnecessary tricks and trusting the model a little more. Every time we simplified the architecture, the system got faster, more stable, and smarter. Because we think context engineering should, the goal of context engineering is to make the model's job simpler, but not harder. So if you take one thing from today, I think it should be build less and understand more. Well, thank you so much, everyone. And thanks again to Lance and the LinkedIn team for having me. Can't wait to see what you guys all build next. Now back to Lance. Yeah, amazing. Thank you for that. So we have a nice set of questions here. Maybe we can just start hitting them and we can kind of reference back to the slides if needed. And Peek, are your slides available to everyone? Oh, yeah. Yeah, I can share the PDF version afterwards. Yes, sounds good. Yeah, well, I'm going to start looking through some of the questions and maybe we can start with the more recent ones first. So how does the alum call the various shell tools? How does it know which tools exist and how to invoke them? Maybe you can explain a little bit about kind of the multi-tier kind of sandboxing setup that you use with Manus. Yeah, I think like imagine you're the person that using a new computer. For example, if you know Linux. you can imagine all the tools are located in slash USR slash bin. Actually, we do two things. First of all, we have a hint in the system prompt telling Manus that, hey, there's a lot of pre-installed command line utilities located in some specific folder. Also, for the most frequently used ones, we already injected in the system prompt, but it's super compact. We do not tell the agent how to use the tools. We only list them. We can tell the agent that you can use the dash dash help. uh flagged safely because all the utilities are developed by our team and they have the same format got it how about um i know you talked a lot about using file system what's your take on using indexing um and do you utilize like do you spin up vector stores on the fly if the context you're working with gets sufficiently large how do you approach that Yeah, I think there's no right and wrong in this space like you've mentioned. But at Manus, we do not use index databases because right now, every sandbox in Manus actually is a new one and users want to interact with things fast. Actually, we don't have the time to build the index on the fly. We're more like plot code. We rely on Grip and Glock. But I think if you consider to build something like more long-term memory. or if you want to integrate some enterprise knowledge base, you still have to rely on that external vector index because it's only about the amount of information that you can access. But for Manus, it operates in a sandbox. And for Coding Agent, you operate in the code base. So it depends on the scale. Yeah, so that's a good follow-up then. So let's say I'm a user. I have my Manus account. I interact with Manus across many sessions. Do you have the notion of memory? Cloud has Cloud MD files, they persist across all the different sessions with Cloud code. How about you guys? How do you handle long-term memory? Actually, in Manus, we have a concept called knowledge, which is explicit memory. For example, every time you can tell Manus, remember every time I ask for something, deliver it in maybe in Excel. It's not automatically inserted into some memory. It will pop up a dialogue and say, here's what I learned from our previous conversation and would you like accept it or reject it so this is the explicit one it requires user confirmation uh but also like we are uh discovering new ways to do it more automatically for example like um a pretty interesting uh thing in agents is that like compared to chatbots user often like correct correct the agent more oftenly for example like a common mistake that managed to make is we're doing like data visualization you know if you're using chinese japanese or korean a lot of time there will be some font issues there'll be arrows in those render visualizations. The user will often say, hey, you should use nodal sense, CGK font. For these kinds of things, the different user will have the same correction. We need to find out a way to leverage these collective feedback. That's like we call it self-improving agent with online learning, but in the parameter free way. Yeah. How about a different question that was raised here and also I think about quite a bit. You mentioned towards the end of your talk that you gained a lot from removing things. And a lot of that is probably because of the fact that also the models are getting better. So model capability is increasing and so you can kind of remove scaffolding over time. How do you think about this? Because this is one of the biggest challenges that I've faced is like over time, the model gets better and I can remove things like certain parts of my scaffolding. So you're building on top of this, the foundation that's like the water's rising. And like, do you revisit your architecture every some number of months with new releases and just delete as the models get better? And how do you approach that problem? Yeah, this is a super good question here because, you know, actually we have already refactored Manus for five times. We've launched Manus in March and now it's October already five times. So we think like you cannot stop because like models are not only improving, but they are changing. models behavior are changed over time like um one way is you can you can work closely with those like model providers but we also have another like internal theory for how we evaluate how we design our agent architecture i've heard a little bit on twitter before it's basically like we all we do not care about a the the static of the performance of a static uh benchmark instead we like we fix the art agent architecture and we switch between models If your architecture can gain a lot from switching from a weaker model to a stronger model, then somehow your architecture is more future-proof because the weaker model tomorrow might be as good as a stronger model today. We think switching between weaker and strong models can give you some early signals of what will happen next year and give you some time to prepare your architecture. For Manus Light, we often do these kind of reveal like every one or two months. And we often like do some like, yeah, do some like research internally using like open source models and maybe like early access to proprietary models to like prepare the next release, like even before the launch of the next model. Yeah. Yeah, it's a good observation. You can actually do testing of your architecture by toggling different models that exist today. Yeah, that makes a lot of sense. What about, best practices or considerations for format for storing data so like markdown files plain text log anything you prefer in particular I think obviously it's yeah how do you think about that kind of file formats yeah I think like it's not about like plain text or markdown but we always like prioritize line based um formats because like it allows like the models to use like grab or like a read from a read from a range of range of lines and also like markdown can sometimes cause some troubles you know um models are trained trained to use markdown really well and sometimes it will maybe for for some model i don't want to say that name but but they often like output too many bullet points if you use markdown too too often yeah so actually we we want to use more plain text yeah makes sense How about on the topic of compaction versus summarization? Let's hit on summarization. This is an interesting one that I've been asked a lot before. How do you prompt to produce good summaries? So for example, summarization, like you said, it's irreversible. So if you don't prompt it properly, you can actually lose information. The best answer I came up with is just tuning your prompt for high recall, but how do you approach this? So summarization, how do you think about prompting for summarization? Yeah, actually, we tried a lot of optimizing the prompt for summarization, but it turns out a simple approach works really well is that you do not use a freeform prompt to let the AI generate everything. Instead, you can define a schema. It's just a form. There's a lot of fields and let the AI to fill them. For example, here are the files that I've modified, and here's the goal of the user. Here's what I left off. If you use this more structured schema, at least... like the output is kind of stable and you can iterate on this. So just do not use like freeform summarizations. Got it. Yeah, that's a great observation. So you structured outputs rather than freeform summarization to enforce certain things are always summarized. Yeah, that makes a lot of sense. How about with compaction then? And actually, I want to make sure I understood that. So with compaction, let's say it's like a search tool. You have the raw search tool output. and that would be your raw message and then the compaction would just be like a file name or something. Is that right? Yeah, it is. It's not only about the tool call, it's also applied to the result of the tool. It's interesting to find out that almost every action in that is just reversible if you can offload it to a. to the file system or an external state. For most of these tasks, you already have a unique identifier for it. For example, for file operations, of course, you have the file path. For browser operations, you have the URL. Even for search actions, you have the query. It's naturally, it's already there. Yeah. That's a great one. I just want to hit that again because I've had this problem a lot. For example, I'm an agent that uses search, I perform, it returns a token heavy tool call. I don't want to return that whole tool message to the agent. I've done things like some kind of summarization of compaction and send the summary back. But how do you approach that? Because you might want all that information to be accessible for the agent for its next decision, but you don't want that huge context block to live inside your message history. So how do you approach that? You could send the whole message back. but then remove it later. That's what Claude does now. You could do a summarization first and send the summary over. You could send everything and then do compaction so that later on, you don't have the whole context in your message history. You only have a link to the file. How do you think about that specifically, if you see what I'm saying? Yeah, I know. Actually, it depends on the scenario. For example, for complex search, it's not just one query. For example, you have multiple queries and you want to gather some important things and drop anything else. In this case, I think we should use sub-agents or internally we call it agent as tool. From the model's perspective, it's still a function maybe called advanced search. It's a function called advanced search. But what it triggers is actually another sub-agent. But that sub-agent is more like a workflow or agentic workflow that has a fixed output schema. and that is the result that returns to the agent. But for other kinds of more simpler search, for example, just searching Google, we just use the full detail format and append it into the context and rely on the compaction thing. But also, we always instruct the model to write down the intermediate insights or key findings into files in case the compaction happens earlier than the model expected. If you do this really well, actually you don't lose a lot of information by compaction because sometimes those old tool calls are irrelevant after time. Yeah, that makes sense. And I like the idea of agent as tool. We do that quite a bit and that is highly effective. But that brings up another interesting point about, and you referenced this a little bit, agent-agent communication. How do you address that? So, Walden Yan from Cognition has a very nice blog post talking about this is a major problem that they have with Devon. So like kind of communication between agents, how do you think about that problem? And yeah, ensuring sufficient information is transferred, but not overloading, like you said, the pre-fill of the sub-agent with too much context. So how do you think about that? Yeah, you know, at Manus we've launched a feature called Wide Research a month ago. Yes, I saw that. Yeah, internally we call it agentic MapReduce because we got inspired from the design of MapReduce. It's special for Manus because there's a full virtual machine behind the session. One way we pass information or pass context from the main agent to sub-agent is by sharing the same sandbox. The file system is there and you can only pass different path here. I think sending information to sub-agent is not that hard. The more complex thing is about how to have the correct output from different agents. What we did here is we have a trick. For every time if the main agent wants to spawn up a new subagent or maybe 10 subagents, you have to let the main agent define the output schema. In the subagent perspective, you have a special tool called Submit Result. We use constraint decoding to ensure that what the subagent submits back to the main agent is the schema that is defined by the main agent. You can imagine that this map reduce operation, it will generate a spreadsheet and the spreadsheet is constrained by the schema. That's an interesting thing that seems to come up a lot with how you design Manus. You use schema and structured outputs, both for summarization and for this agent-agent communication. It's like use schemas as contracts between agent-sub-agent or between a tool and your agent. to ensure that sufficient information is passed in a structured way, in a complete way. Like we're doing summarization, you use a schema as well. OK, fantastic. This is very, very, very helpful. I'm poking around some other interesting questions here. Any thoughts on models? I think you guys use Anthropic, but do you work with open models? Do you do fine tuning? You talk a lot about working with KVCache, so for that, maybe using open models. How do you think about model choice? Yeah, actually right now we don't use any like open source model right now because I think it's not about quality. It's interestingly, it's about cost. You know, we often think that open source model can lower the cost, but if you're at the scale of Manus and if you're building a real agent, which the input is way longer than the output, then KVCache is super important. And distributed KVCache is very hard to implement if you use like open source solutions. And if you use those frontier LN providers, they have more solid infrastructure for distributed cash globally. Sometimes if you do the math, at least for Manus, we find out that using these flagship models can sometimes be even more cheaper than using open source models. Right now, we're not only using Anthropics. Of course, Anthropics model is the best choice for agentic tasks, but we're also seeing good progress in Gemini and in OpenAI's new model. I think right now, these frontier labs are not converging in directions. For example, if you're doing coding, of course, you should use Claude. If you want to do more or multimodality things, you should use Gemini and OpenAI's model is super good at like complex math and reasoning. So I think for application companies like us, one of our advantages is that we do not have to build on top of only one model. You can do some like task level routing or maybe even subtasks or step level routing if you can like calculate like if you can. can pull her in that kind of KVCache validation. So I think it's a vantage for us. And we do a lot of evaluations internally to know which models to use for which sub-task. Yeah, that makes a lot of sense. I want to clarify one little thing. So with KVCache, so what specific features from the, or what from the providers are using for cache management? So, okay. I know like Anthropic has input caching as an example. Yeah. That's what you mean. Okay. Got it. Yeah. Cool. Okay. Perfect. Cool. I'm just looking through some of the other questions. Yeah, tool selection is a good one. Right. So you're talking about this. You don't use like indexing of tool descriptions and fetching tools in the fly based on semantic similarity. How do you handle that? Like what's the threshold for too many tools? Yeah, cool choices, a classic. How do you think about that? Yeah. First of all, it depends on the model. capacity for tools. But I think a rule of thumb is probably not to include more than 30 tools. It's just a random number in my mind. But actually, I think if you're building a, we call it a general AI agent like Manus, you want to make sure those native functions are super atomic. So actually, there are not that much atomic function that we need to put inside the action space. So for Manus, right now, we only have like... 10 or 20 atomic function and everything else is in the sandbox. Yeah. So we don't have to pull things dynamically. Yeah. Good point. Actually, let's explain that a little bit more. So you have, let's say, 10 tools that can be called directly by the agent. But then I guess it's like you said, the agent can also choose to write a script and then execute a script. That expands its action space hugely without giving it like. You don't have an independent tool for each possible script, of course, that's insane. So a very general tool to write a script and then run it does a lot. Is that what you mean? Yeah, exactly. Why we are super confident to call Manus a general agent because it runs on a computer and computer are turning complete. The computer is the best invention of human. Theoretically, an agent can do anything that maybe a junior intern can do. using a computer. So with the shell tool and the text editor, we think it's already Turing complete. So you can offer a lot of things to a sandbox. Yeah, okay. That makes a lot of sense, right. And then how does Manus, so is, are all, so okay, maybe I'll back up. You mentioned Kodak. With Kodak, my understanding is the model will actually always produce a script and that'll then be run inside a code sandbox. Every tool call is effectively a script is generated and run. It sounds like you do some hybrid where sometimes a mall can just call tools directly, but other times it can actually choose to do something in the sandbox. Is that right? It's a hybrid approach. Yeah. I think this is super important because actually we try entirely to use Kodak for Manus, but the problem is if you're using code, you cannot leverage constrain decoding. and things can go wrong. Yeah. But Kodak has some special use cases, as I mentioned earlier in slides. For example, processing a large amount of data, you don't have to pour everything in the tool result. Instead, you put it inside maybe the runtime memory of Python, and you only get the result back to the model. We think you should do it in a hybrid way. Got it. Allow for tool calling and you have some number of tools, maybe 10 or something that are just called directly, some number of tools that actually run in the sandbox itself. Yes. Perfect. That makes a ton of sense. Very interesting. Then maybe how do you keep a reference of all the previously, I guess you have, so you basically will generate a bunch of files. Oh, actually, sorry. Maybe I'll talk about something else. How about planning? Tell me about planning. I know Manish has this to-do tool or it generates a to-do list and start of tasks. Yeah. Tell me about that. Yeah. I think this is very interesting because like. At the beginning, Manus uses that to-do.md paradigm. Like, it's kind of... I don't want you... use the word stupid, but actually it wastes a lot of turn. Back in maybe March or April, if you check the log of some Menace task, maybe one-third of the action is about updating the to-do list. It wastes a lot of tokens. Yeah. Right now, we're using a more structuralized planning. For example, if you use Menace, there's a planner at the bottom of the system. Internally, it's also a tool call. We implemented using the agent as tool paradigm. so that like there's a separate agent that that is managing the plan so actually right now the latest version of manage we are no longer using that to do that md thing of course like to do that md still works and it can generate like good results but if you want to say say tokens you can find another way got it yeah so you have like a planner agent and it's more like for a subtask it'll be more like agent as tool call type things yeah and you know it's very important to have a separate agent to that has a different perspective. It can do some external reviews and you can use different models for planning. For example, sometimes Rock can generate some very interesting insights. That's a great one actually. Think about multi-agent then. How do you think about that? You might have a planning agent with its own context window, makes a plan, produces some plan object, maybe it's a file, or maybe it just calls sub-agents directly. How do you think about that? How many different sub-agents do you typically recommend using? I think this is also depends on your design, but here at Manus, actually Manus is not the typical multi-agent system. For example, we've seen a lot of different agents that divides by role. For example, you have a designer agent or programming agent, manager agent. We don't do that because we think why we have this is because this is how human company works. This is due to the limitation of human context. So in Manus, Manus is a multi-agent system, but we do not divide by role. We only have very few agents. For example, we have a huge like general executor agent and a planner agent and a knowledge management agent and maybe like some, yeah, data API registration agent. Yeah. So we are very, very cautious about adding more sub-agents because of the reason that we've mentioned before, the communication is very hard. And we implement more kinds of like sub-agents as agents as tools, as we mentioned before. Yeah, that's a... Yeah, that's a great point. I see this mistake a lot, or I don't know if it's a mistake, but you see anthropomorphizing agents a lot. Like, it's my designer agent, and I think it's kind of a forced analogy to think about, like, a human org chart in your subagent. So, got it. So, for you, it's like a planner, a knowledge manager, and knowledge manager might do what? Like, what would be the task of a knowledge manager? Yeah, it's even more simple. As we mentioned, we have a knowledge system and manage. What the knowledge agent does is that it reviews the conversation between the user and the agent and figure out what should be saved in the long-term memory. So it's that simple. Got it. Yeah, okay, got it. It's like a memory manager planner. And then you have sub-agents that can just take on, like a general executor sub-agent that can just call all the tools or actions in the sandbox. That makes sense. Keep it simple. I like that a lot, right? That makes a lot of sense. Yeah. Yeah, let me see if there's a bunch of questions here. But we did hit a lot, so that's actually... How about guard railing? Someone asked a question about kind of safety and guard railing. How do you think about this? I guess that's the nice thing about a sandbox, but tell me a little bit about that, how you think about it. Yeah, I think this is very... a very sensitive question because like you know if you have a sandbox that's connected to the internet everything is dangerous yeah so we put a lot of effort like in guard railing like at least we do not let the information to get out of the sandbox for example like if you like got prompt injected we have some like uh checks on like outgoing traffic for example like we'll ensure that no like token things will go out of the sandbox and if the the user wants to like print something out of sandbox we have those kind of like like like what we call it? Removing things and to ensure that no information goes out of the sandbox. But you know, for another kind of thing is that we have a browser inside of Manus and the browser is very complicated. For example, like if you log into some like your websites, you can choose to let Manus to persist your login state. And this turns out to be like very tricky because like sometimes the content of the webpage can also be like malicious. Maybe they're doing like like prompt injection and this i think is somehow like out of the scope for an application company so we're moving uh we're working very closely with those computer use model provider for example like anthropic google yeah they're adding a lot of guardrails here so right now in manis every time you do some like sensitive operations whether or inside the um the browser or in the sandbox management will require a manual confirmation and you must accept it or otherwise you have to take over it to finish it yourself so i think like it's pretty hard for us to like design a a very well-designed solution. It's a progressive approach. So right now, we're letting the user to take over more frequently, but if the guide rail itself in the model gets better, we can do less. Yeah, yeah. What about the topic of evals? This has been discussed a lot, quite a bit online. If you've probably seen, you know, Cloud Code, they talked a lot about just doing less formal evals, at least for code, because code evals are more or less saturated, lots of internal dogfooding. How do you think about evals? Are they useful? What evals are actually useful? What's your approach? Yeah. You know, at the beginning, at the launch of Manus, we're using public academic benchmarks like Gaia. But then after launching into the public, we find out that it's super misaligned. Models that get high scores like Gaia, the user don't like it. So right now, we have three different kinds of evaluations. First of all, most importantly is that for every completed session in Manus, we'll request the user to give a feedback. We give one to five stars. Yeah, this is the gold standard. We always care about the average user rating. This is number one. And number two, we're still using some internal automated tests with verifiable results. For example, we have created our own data set with clear answers. But also, we still use a lot of public academic benchmarks, but we also created some some data sets that's more focused on execution because like most benchmark out there are more about like read-only tasks so we designed some like like um like executing tasks or transactional tasks because we have a sandbox we can like frequently reset the test environment so these are the automated parts and most importantly number number three we have a lot of interns, you know, you have to get a lot of real human interns to do like, like, evaluations on things like website generation, or data visualization, because like, it's very hard to design a good reward model that knows whether the output is visually appealing. Like, it's about the taste. Yeah, so we still rely on a lot of a lot. Yeah. Perfect. Yeah, let me ask you, I know you're coming up on time, but I do want to ask you about this emerging trend of of reinforcement learning with their viable rewards versus just building tool calling agents so like cloud code extremely good and they have the benefit because they built the harness and they can perform rl on their harness and it can get really really good with the tools they provide in the harness do you guys do rl um or how do you think about that because of course in that case you would have to give you some open models i've been playing with this quite a bit lately How do you think about that, just like using Toolkong out of the box with model providers versus doing RL yourself inside your environment with your harness? Yeah, how do you think about that? Yeah, I mentioned before starting Madness, I was kind of a model training guy. I've been doing free training, post-training, RL for a lot of years. But I have to say that right now, if you have sufficient resource, you can try. But actually, as I mentioned earlier, MCP is a big changer here. because like if you want to support mcp you're not using a fixed action space and if it's not a fixed action space it's very very hard to design a good like reward and you cannot generate a lot of like the the rollouts and feedbacks will be unbalanced so if you want to build a model using like that supports mcp you are literally building a foundation model by yourself so i think like every everyone in the in the community like model companies they're doing the same thing they're doing the same thing for you. So right now, I don't think we should spend that much time on doing RL right now. But as I mentioned earlier, we are just discovering new ways to do, maybe you call it personalization or some sort of online learning, but using a parameter-free way, for example, collective feedbacks. Yeah. One little one along those lines is, is it the case that, for example, Anthropics done reinforcement learning and verified rewards on some set of tools used within Cloud Code? Have you found that you can kind of mock your harness to use similar tool names to kind of unlock the same capability, if that makes sense? Like, for example, like, like, I believe they've just, you know, they've obviously performed, you know, they utilize Glob, uses grep, uses some other set of tools for manipulating the file system. Can you effectively reproduce that same functionality by having the exact same tools with same tool names, same descriptions in your harness? or kind of how do you think about that like unlocking um unlocking the yeah right you see what i'm saying yeah yeah i know the clear answer here but for us we actually try not to use the same name because like it was like if you design your own function you maybe have like different requirements for that function and the parameters of the input arguments might be different so you don't want to like confuse the model like if the model is trained on a lot of like post-training data that has some internal tools. You don't want to let the models be confused. Okay, got it. Perfect. Well, I think we're actually at time. I don't want to respect your time. I know it's early. You're in Singapore. It's very early for you. Well, this was really good. Thank you. We'll definitely make sure this recording is available. We'll make sure slides are available. Any parting things you want to mention, things you want to call out, calls to action? Yeah, people should go use Manus, but the floor is yours. Yeah, I just want to say, everybody try Manus. We have a free tier. Yeah, absolutely. Hey, thanks a lot, Pete. I'd love to do this again sometime. Yeah, thanks for having me. Yep. Okay, bye. Bye.

Join us for a deep dive into context engineering – the critical practice that determines how well your AI agents perform in production. Lance Martin from LangChain and Manus co-founder Yichao "Peak" Ji share battle-tested strategies for managing context windows, optimizing performance, and building agents that scale. Peak was recently named one of MIT's Innovators Under 35 for his work on AI agents. Here, we cover Manus's context engineering approach.

Strategies include:

(1) **Context reduction** via dual-form tool results (full/compact) with policy-based compaction and schema-driven summarization
(2) **Context offloading** through layered action spaces (function calling → sandbox utils → packages/APIs) with filesystem-based state management and shell utilities instead of vectorstore indexing
(3) **Context isolation** using minimal sub-agents (planner, knowledge manager, executor) with agent-as-tool paradigm and constrained decoding for schema-based inter-agent communication.

📊 Access the Presentations:

Lance Martin's slides (LangChain): https://docs.google.com/presentation/d/16aaXLu40GugY-kOpqDU4e-S0hD1FmHcNyF0rRRnb1OU/edit?slide=id.p#slide=id.p 

Yichao "Peak" Ji's slides (Manus): https://drive.google.com/file/d/1QGJ-BrdiTGslS71sYH4OJoidsry3Ps9g/view?usp=sharing

Ready to start building reliable agents? 
Sign up for LangSmith, our agent observability & evals platform: https://www.langchain.com/langsmith/?utm_medium=social&utm_source=youtube&utm_campaign=q4-2025_meetup-manus_co

Learn how to observe and evaluate agents on LangChain Academy:
https://academy.langchain.com/courses/quickstart-langsmith-essentials/?utm_medium=social&utm_source=youtube&utm_campaign=q4-2025_youtube-academy-links_aw

Chapters
0:01:00 Introduction to context engineering
0:12:00 Why context engineering in Manus
0:15:00 Context reduction in Manus
0:19:20 Context isolation in Manus
0:22:17 Context offloading in Manus
0:29:00 Avoid context over-engineering
0:31:00 Q&A: Explain sandbox utils in Manus
0:31:55 Q&A: Indexing (vectorstore) vs just using files
0:32:50 Q&A: Memory in Manus
0:34:30 Q&A: Manus and The Bitter Lesson
0:36:44 Q&A: Data format
0:37:45 Q&A: Summarization tips
0:40:00 Q&A: Sub-agents as tools
0:43:57 Q&A: Model choice
0:46:20 Q&A: Tool selection
0:49:48 Q&A: Planning
0:53:35 Q&A: Guardrails
0:55:39 Q&A: Evals
0:57:15 Q&A: Using RL