From Context Engineering to AI Agent Harnesses: The New Software Discipline

There's been a democratization shift in the industry. And then because of that shift, most people using MLAI today are working at a higher level of abstraction. So rather than intense focus on model architecture and training, most users now just are headed this object, this extremely powerful new computing primitive. And what do I do with this? And this is where I've been operating for the last few years at Langchain, building on top of these LLMs. prompt engineering, context engineering, fine tuning, building agents, all of these new disciplines built on top of this new primitive that's being offered through an API by a small number of players. So I think those are a few major shifts that we've seen in the landscape over the last few years. That was Lance Martin, a machine learning engineer at Langchain, outlining a key shift in the generative AI era. We're moving from the challenge of training models to the new engineering discipline of orchestrating them at scale. Lance used to build and scale production ML systems at Uber, including self-driving technology, and now builds tools at Langchain to help teams across all verticals build and deploy AI-powered applications. In this episode, Duncan Gilchrist and I speak with Lance about what's changed since the early ML days and how core principles like simplicity and observability must be adapted for non-deterministic systems. Lance brings so much insight from the bleeding edge of the space with examples from Claude Code and Manners discussing the practical disciplines of context engineering including context rot or why the effective context window is often much smaller than the token limit and the three-part playbook to manage it reduce offload and isolate This also includes using multi-agent architectures for context isolation. We also cover the emerging architecture of the agent harness, which manages tool calls, essentially how LLMs can do things, and how builders can use only a few atomic tools like a bash tool to expand the agent's action space dramatically. This episode is a deep dive into engineering discipline and also gives technical leaders clear insight into How teams are building and delivering value at the bleeding edge where the foundation models powering your applications are constantly and exponentially improving. High Signal is brought to you by Delphina, the AI agent for data science and analytics. If you enjoy these conversations, please leave us a review. Give us five stars, subscribe in the newsletter, and share it with your friends. Links are in the show notes. I'm Hugo Bowne-Anderson. Welcome to High Signal. Let's jump in. Hey there, Lance, and welcome to the show. It's great to be here. And I've known Duncan for many years. It's a pleasure to be on and great to meet you as well. Totally. And look, I'm so excited to hear about what's happening at Langchain and all the wonderful things that you're affording people to do there. But what I'm also really excited about is you've worked on production ML systems at places like Uber prior to working on generative AI tooling at Langchain. So you have a wonderful perspective on what's changed. So I'm wondering if we could open by letting us know what feels fundamentally different about building and maintaining ML systems versus what people are doing now with generative AI and LLMs in particular. That's right. Yeah. So, you know, Duncan and I overlapped at Uber. This was back in 2015, 2016 era. And I think there's been a few interesting shifts in the ML AI landscape since that time. I think one is architectural consolidation. So we saw the emergence of the transformer architecture, extremely expressive. We saw driven by scaling laws and compute data model size models get much bigger we saw other architectures like cnn's rn's that are a little bit more specialized kind of get swallowed by transformers so we had architectural consolidation and scaling laws driving much much larger models that's thread one and then thread two was i worked in self-driving for a number of years at uber than after uber and in that era there was approximately the same amount of orgs that were training models were housing or using them. So it was one-to-one in the sense that each self-driving company was trying to train their own models. It was highly proprietary, a lot of in-house expertise at really all these companies. And I think obviously beyond self-driving where ML is being deployed in recommender systems, typically organizations that were using ML were also training the models. Now that's entirely flipped. You've had the emergence of a small number of foundation model providers, models have become extremely large. Most people using AI today are not actually training models. So that's kind of this, there's been a democratization shift in the industry. So architectural consolidation, democratization shift. And then because of that shift, most people using MLAI today are working at a higher level of abstraction. So rather than intense focus on model architecture and training, most users now just are handed this object, this extremely powerful new computing primitive. And what do I do with this? And this is where I've been operating for the last few years at Langchain, building on top of these LLMs, prompt engineering, context engineering, fine tuning, building agents, all of these new disciplines built on top of this new primitive that's being offered through an API by a small number of players. So I think those are a few major shifts that we've seen in the landscape over the last few years. I think that kind of contrasts from... classic or traditional ML to Gen AI is so interesting. And you and I have a lot of war wounds, I think, from and battle scars from the Uber days. I'd love to explore with you like which lessons from traditional ML still actually apply in Gen AI systems and which start to fall apart. Yeah, so that's an interesting one. And what I would say here is actually, even though we're now handed these incredibly powerful models through, for example, APIs offered by these frontier labs, simplicity remains essential. I think it is very important to start with the simplest possible solution. We see many organizations that we work with, the Langchain and others that I've consulted with, will jump to agents are in the air, agents are a buzzword, I want to build an agent. And I think we'll talk about this more in detail later. But really thinking through the problem you're trying to solve, there's many different ways to do it. Prompt engineering, a simple workflow, building an agent with simple content engineering, all the way up to maybe building an agent with some kind of... RL and loop of reinforcement fine tuning. There's a spectrum of solutions you can use with these models depending on your problem. And I think starting simple is very important. The other thing that's really critical is actually observability evaluation. So you can build an agent, but actually having the ability to understand what's happening and to actually evaluate it in a rigorous way is obviously extremely important. So at Langtron, we do a lot of work on observability and tracing and also a test suite and evaluation suite. Beyond just simple unit tests, I think many software orgs are familiar with just simple unit tests. Working with ML systems, particularly LLMs, which are non-deterministic, needs a new kind of evaluation, which we could talk about later. I think the final point is a really interesting observation I heard from Jason Wei, who is at OpenAI for many years and now MSL, MetaSuperintelligence Lab. He talks about this idea of verifier's law, which says that the ability to train an AI to solve a task is proportional to how easily verifiable a task is. So verification means coding. You can just compile the code. You can run it, make sure it runs. That's kind of what verification means. And tasks that are easier to verify are actually easier to, for example, apply reinforcement fine-tuning to. That's what he's referring to. So actually, setting up evaluation, and part of evaluation is establishing some verification metric, is actually very helpful and important foundation if you ever want to apply reinforcement fine-tuning or train fine-tune models. for particular tasks so setting up evaluations with clear verification or criteria is very important for quality and also if you ever want to move into fine-tuning or more advanced things so those are three things that are still very important today with these new systems which were always of course important for in the prior eras of ml there's so much meat and insight in there that we will get into throughout this conversation one thing that i do think we will talk about is how we go about building and evaluating agents and it does feel kind of like a two cultures thing in some ways because as you've said starting and building small is incredibly important to be able to inspect evaluate and deliver value and yet we'll talk about a blog post you wrote about context engineering in manis typical manis call has 50 tool calls anthropic has told us to start slow with workflows and yet they also have published about their multi-agent research system which is a big sprawling behemoth in in a lot of ways i am interested in before we we get to that point to talk about a wonderful post of yours that we'll link to in the show notes called learning the bitter lesson and you quote rich sudden by saying general methods that leverage computation are ultimately the most effective and i'm wondering if you could expand on this quote unquote bitter lesson and tell us how it's shaped the way you approach designing systems today yeah actually i think this is one of the most interesting challenges associated with now building on top of this new kind of lm compute primitive that we all have access to so basically rich sutton 2019 put out this very important like kind of seminal essay called the better lesson and the intro line follow goes the biggest lesson that can be read from 70 years of ai research is that general methods that leverage computation are ultimately the most effective and by a large margin. So simple methods throw more compute, often beat more complex methods with more kind of human biases. And like a classic example of this is in vision, we had convolutional neural networks. They encoded various inductive biases about how we should solve vision paths, a classification or object detection. Transformers ultimately became state of the art for that particular task. Transformers are a very general message passing architecture. And so The observation was just a basic architecture like transformers plus more data and scale actually ultimately beats more kind of handcrafted methods. That's been the trend we've seen in AI for over 70 years. Now, the link here is that also applies to things that we're building on top of LLMs. So it also applies at the kind of AI engineering layer. This is one of the biggest lessons I've learned in building LLM applications. Because here's the problem. Just as the model layer, we're building on top of exponentially We're designing architectures on top of kind of exponentially rising or increasing compute. We're also now building applications on top of exponentially improving LLMs or models. And so what you build today and the assumptions baked into your architecture in terms of whatever app you're building will not be correct in six months when a new model is out there. It's much, much better. And I saw this play out in my own work on this project called Open Deep Research. And that's what I talk about in the blog. And I just cover a year of working on this project. It's basically a fully open source deep research agent. I started it as a very simple workflow back in 2024 because basically tool calling with LLMs was weak actually. It didn't really perform that well. I didn't want to, and we'll talk about what agents are versus workflows a bit later. Suffice to say, I started with a particular architecture that was not using an agent. And over time, as tool calling got much better, I basically had to re-architect OpenDeep Research three or four times to keep up with increasingly improving models. I talked to Manus about this in a webinar we did about two weeks ago. They mentioned that Manus is one of the most popular general purpose agent products out there today. It's based in Singapore. They actually re-architected Manus five times since launching in March. So what you're seeing across the industry is building on top of this new primitive that's getting much, much better all the time, forced you to continually re-architect. reassess your assumptions and rebuild your applications. Another great example of this, Boris Cherney from Cloud Code mentioned in passing in one of his talks, like in the Q&A, the kind of secret sauce and so forth, the kind of secret sauce of Cloud Code is, he said, 70% model, 30% kind of scaffolding or harness, right? They have a no-agent harness that they use with Cloud Code. And the follow-up question was, oh, okay, but over time, this model gets better. Does that mean all your work on the harness is like irrelevant? He said, yeah. He said, yeah, that is the case. So what's happening is over time models get better and you're having to strip away structure, remove assumptions and make your harness or your system simpler and adapt to the models. And so this is something that's very hard. You can't just have a fixed architecture, fixed scaffold, fixed harness and be done because you're building on top of something that's always improving. I think this is one of the biggest lessons that is tricky. And I think we'll talk about a little bit more later, but that's one of the biggest things I've observed in working with LLMs. It's embracing change. being willing to re-architect your system. And the nice thing is with LMs being extremely strong at, for example, Code Assist, Cloud Code, Cursor, Devon, it's very easy to rebuild things. But embracing that fact that, look, you're building on top of a kind of primitive that's always getting better, you have to constantly re-architect your application and you can't be shy about that. I think it's one of the most disorienting things, though, for people who are starting to work with LMs for the first time. Would you mind just clarifying what you mean by a harness as well? Yeah, right. when you build an agent, and we'll talk about agents and workflows in more detail, but basically when you build an agent, you have the LLM. So LLM is exposed through some kind of SDK. For example, you could be using Cloud, you could be using OpenAI. But when you build an agent, as an example, you're taking that LLM SDK, you're binding a bunch of tools to it, and you're basically allowing that LLM to call tools. So when it calls a tool, it basically just produces a structured output that adheres to whatever tool you provided. Let's say it's a search tool. The search tool takes a single parameter query. The LLM will produce a function call or tool call that just has like query and then whatever query parameters that you want. That's all tool call is. You need something that actually executes that tool call. And that's kind of where the harness comes in. So the harness will actually do that and other things. We'll talk about context engineering in a bit. But the harness is the thing that actually says, okay, the LLM actually made this tool call. Okay, I'll go ahead and run the tool. I'll take that tool result, package it as a message. add it to, for example, typically you have a message list that's growing that you're passing back to the LLM every turn of your agent. So managing that message list, managing tool execution, packaging up the tool results as messages, passing them back to the LLM, that's the harness. And we'll talk about it in a little bit, but actually it handles more than it typically handles some kind of logic or what we might call context engineering. And Cloud Code, for example, has a very interesting harness. You actually can see when it's running, you can see the tool calls it's making. Like if you ever work with cloud code, you can see like it'll run bash tool or run different search tools. You actually see that in the trace as it's running. So the harness is doing all that under the hood and passing those results to the model and model's reasoning, making additional tool calls and so forth. A few times you've used words like workflows and pipelines and agents. Can we unpack those a little bit? What do those actually mean? How do they fit together? And where are they in the generative kind of hype cycle? Yeah, this is a really good one to clarify. So I've said the word agent a few times. I might have said the word workflows. We should break all these things down carefully. So actually, probably the best blog post, I'm sure we'll put in the show notes, is from Anthropic on this. Came out late last year. It's called Building Effective Agents. So they define workflows as systems where LLMs and tools are orchestrated through predefined code paths. It follows a predetermined sequence. And you can have LLM calls embedded in that sequence. But you have an application that goes A to B to C to D. Every time you run it, step C could be an LLM call to do something. That's a workflow. An agent's a bit different. An agent's a system where an LLM, they say, dynamically direct its own processes and tool usage, meaning control over how it accomplishes tasks. So what does this actually mean? In practice, all it means is I have an LLM. I have some set of tools. I bind the tools to the LLM. Let's say I have tools A, B, C, D. The LLM can call those tools in any order it wants to solve the problem. it can call BCD. Whereas in a workflow, you lay out very precisely the steps, A, B, C, D. So that's the key difference. Workflows follow some predefined set of steps that I lay out as a developer that can involve LLM calls. Whereas agents allow an LLM to call tools autonomously in a loop in any order they see fit. And I think the key point to make is the difference is autonomy. Agents are really good for tasks that you can't really enumerate ahead of time. Research is a classic. That's why deep research is one of the seminal agent products. Research is open-ended. The next step is conditioned on the prior one. I'm going to do a search, get some results, reason about the results, and do another search. Whereas tests, I want to run this test suite, is a classic workflow type problem. Every time a PR is put up, I want to run these five tests. That's more of a workflow thing. So it sounds like you advocate for structured workflows. They're simpler in certain kinds of cases. How do you think about what kinds of problems are better solved by those versus more agentic approaches? Yeah, so this is one of the classic, this is one of the classics. Okay, so when to use workflows, when to use agents. I'll share some nice documentation in the show notes on this, but there's a few different resources I like. One in particular is a talk given by a guy from Shopify who developed what they call Roast. So Roast is a framework built internally at Shopify for laying out workflows. It's very similar to a framework that we have at Langchain called LangGraph. LangGraph is an extremely popular framework for building agents or workflows. But I really like the Shopify example because it really exemplifies a lot of the rationale and reasons why we built LangGraph. Workflows are great when you have problems that have predefined predictable steps. Migrating a legacy code base. Running some set of tests. The Shopify talk mentions a lot of those are some of the things that they use to motivate Roast. Very well defined. predictable steps. Two is consistency and repeatability. So when you need deterministic behavior and clear oversight, like testing is another great example of that. With every PR you want these end tests run. And I guess known sequences, steps A, B, C, D. That's really where workflows shine. And then agents are good for anything that requires ongoing adaptation, debugging, iteration. Research is a classic. Coding is another good one. That's why coding agents are so popular. Solving problems with coding often are iterative. You try one solution, you might run through a set of tests fail, you try again. So more open-ended, adaptable problems are much better for agents. Research, coding, predictable, well-defined steps, migrations, tests, much better as workflows. And one nuance is actually you can embed agents in workflows. So you get a workflow of end steps, and one of those steps could be calling an agent to do a thing. And in fact, Shopify talks about that a lot in their roast talk, which I'll be sure to link. So I think it's a subtle point that they actually can play together. But it is also true that lots of problems that people want to solve with agents, you could absolutely solve just by laying out a workflow. The other subtle point that I think is worth expanding on slightly, because it trips up a lot of people, is that it's not agent or not. And when we talk about workflows, a lot of people will use the term agent or agentic to describe that. And of course, Anthropic makes that clear in the blog post you mentioned from last December. which is there's a spectrum of agency where maybe you've got an LLM and you're adding some memory, a couple of tool calls, retrieval, and then you're starting to build workflows. So it isn't an on switch or an off switch. The other thing I think worth mentioning, I think you're speaking to this already, is having high agency in your software in terms of having an agent with incredibly high agency works very well when you have... relatively strong supervision as well. So a human in the loop with it who can guide it and train it and have conversations with them, maybe even, God forbid, check the code it's written, for example. So I think this is an interesting point. Part of the reason why a lot of people were hesitant to build agents prior to, for example, this year is there's three different reasons. One being that tool calling was not as reliable. But as a consequence, you had to babysit agents much more. You had to check their work very carefully. They get caught in what you might call a tool calling loop. So continually trying to call the same tool many times, that just burns tokens needlessly. So as LMs have gotten better at tool calling, it's become more and more feasible to build agents that are actually effective, that don't fall into these kind of common traps and failure modes, as you mentioned, that require a bit less babysitting. Now I do want to mention, it is still true though that agents, because they can autonomously call tools, do pair very well with Hume in the loop. And there may be certain tools, for example, with Cloud Code, you can basically, you can run it in safe mode where basically you approve certain tool calls. And that's true across many different agentic systems. That's a whole topic we get into, but sandboxing agents is often very important and so forth. So because they're running autonomously, you do indeed have to be quite careful both about token usage. and then spinning off into kind of very kind of long sequences of tool calls and also security so making sure they don't make tool calls that you don't want them to make hit sensitive systems delete things and so forth so it is absolutely true that because agents have higher autonomy or agency we often have to be a little bit more careful about what they're doing and sandbox them appropriately Absolutely. And Anthropic actually made that very clear when they first released their first prototype for Claude Code. I think they're like, please do sandbox this. This is highly experimental. Yes. And then like earlier this year, we saw even people like Steve Yeager say on Twitter, I can finally talk about this now. I deleted a production database by Vibe Coding. And the opposite of safe mode. Kurser used to call it YOLO mode. They don't anymore. They've changed the name. I know. But you actually spoke to a really interesting point, which is... Models are getting significantly better. So can you tell us a bit about how the improvement or more affordances of models have made building agents or made agents more reliable at what they do? You know, it's there's a few different interesting threads here. So one is I'll link this in the notes, but so meter publishes a kind of a leaderboard or kind of an evaluation that measures the length of tasks that LMS can accomplish. And I have to go back and check. I believe it's doubling every seven months. So this is like the length of human equivalent work. So it's something like agents can at a 50% success rate accomplish tasks that take a human two hours a day or something. There's a bunch of different models evaluate. But the point is the kind of autonomy level of LLMs is doubling every seven months. It's one of the interesting scaling laws to track. So that's a consequence of models getting better at tool calling. Largely that. Getting better at tool calling. allowing them to perform longer horizon tasks. So it really comes down to the fact that these models are indeed getting quite a bit better at instruction following and tool calling. And also I would note, you see this with Cloud Code, they're getting better at adapting. So for example, they do make an error. For example, they format a tool calling correctly. They can see that, for example, that trace and they can correct. So self-correction is another very important point. So it's really all these things coming together. At the end of the day, models getting much better at tool calling. allowing for longer horizon tasks. That's really the key driver here. And that's something, the self-healing is something Anthropic has published quite a bit about in their building their multi-agent research system. And we do have a bunch of other topics to get to, but something you've actually been speaking to implicitly is the rise of background agents, right? So as the models get better at doing tool calls, the ability for us to send them off for longer. And so I'm wondering if you can tell us just what you're seeing with respect to this burgeoning field of background agents. It's funny, at least at Langton, we call them ambient agents. We have a whole course on it. So I did a whole course on building ambient agents in Langrath. And the use case I built in the course that you can build up to is an agent that will run your email. And so it just runs autonomously every night on a cron and it'll process all your emails. Actually, sorry, it's not every night. You can run it. It actually pings every 10 minutes. So it's running the background every 10 minutes constantly. You can configure that any way you want though. You could have it run once a day overnight. For me, it's every 10 minutes and it's constantly kind of monitoring your emails. It pulls them in, it triages them. It decides which ones to respond to, which not to respond to. And it'll produce responses and queue them all up for you. You approve them through a little interface we built and it'll fire them all off. So Harrison or CEO actually uses it. I don't cause I don't get that many emails, but so that's a good example of an ambient agent. I think it's a great point. It's a very good emerging form factor. Codex is a great example for code. Just kick it off. It runs async, does stuff for you. It makes a lot of sense. The catch, I would say, is that in the context of coding, it can actually increase the burden, the review burden. So, for example, you have to really trust the system to do a bunch of work autonomously and come back to you after some period of time. If it spins off on a task for a long period of time and it's on the wrong track, you get this big body of work done at the end and you say, oof. So designing the right kind of guardrails or human loop, oh, it's stuck. How do you check it? How do you prove what it's doing is a little bit tricky still. Like with my email agent, I have a few different gates. Like basically it pings me if it needs to ask a question. It pings me when it's going to prepare a response, like an email that it's drafted. And it'll ping me if it just decides this is worth ignoring to let me confirm. Long story short, when you're working these async or ambient or background agents, you do have to be careful to design the system such that it has the right kind of human loop checkpoints. Because if it just goes off and does a bunch of work behind the scenes, that can be problematic for all the obvious reasons. So that's a little bit of the trick to these async ambient agents. I personally, for code, I actually do use cloud code synchronously mostly. That's just me. And I do use async agents for things like email. Or at least I've built it and I've used it for a while, but Harrison continues to use it. I just don't get enough email to justify it. But I do think with async or ambient agents, being very thoughtful about how you set up human loop is very important. and how much trust you can build in the system. I'll make a small note here that in the email example, a very important aspect of AmbientAge, I think, is memory. Because you want them to handle long horizon tasks autonomously. So ideally, they remember your preferences because you're endowing them with longer horizon work. So in my little email example, I actually have a memory system. So every time I give it feedback, it records my feedback and bakes it into this little long-term memory, which is just stored in a very simple set of files. And those files are updated constantly as I give it feedback, so it gets smarter and smarter over time. So it runs autonomously, but it's also learning my preferences. Because it would be very annoying if these systems run autonomously but don't learn from our feedback and just keep making the same mistakes over and over. I think that's another tricky thing about autonomous agents, though. Ideally, they have some form of memory so they can adapt and learn. Zooming one click on that actually, I think the context you feed into the agent is obviously so critical to making the agent work correctly. And you've actually written a lot about this term kind of context engineering. I'd love for you as a domain expert to maybe define that term and help us think through why it's important and how leaders should think about it. Yeah, so this is actually a very important term that's emerged in the last couple of months. So the way I think about it is, let's say I build an agent. I take an LLM, I bind some tools. Let's say it's a deep research agent. I actually made this mistake, so I'll walk through it exactly. You bind up onto search tools, you have it run. It does a search, returns the results. Those results are appended to a message list. You pass those messages back to the LLM, makes another search tool call. Same thing. That happens five or six times. By the end, that message list could be quite big, depending upon how large or how many tokens are in each search result. You could be talking, in my case, hundreds of thousands of tokens so extremely token heavy if you're just doing this kind of naive tool calling a loop thing which is a base case agent and that's extremely expensive and slow so actually being very thoughtful about the context you basically feed to your llm is important not just for latency and cost but also chroma i'll link this in the show notes put out a really nice report on what they call context rot so basically as context gets longer performance degrades And Anthropic mentions it recently in a nice post they have on context engineering. And they mentioned the attention mechanism starts to degrade with respect to context length. So the point is, agents in a naive form use a lot of tokens. If it's just tool calling a loop, depending on the tool calls you're using, oftentimes, for example, with a search tool, it's pretty token heavy. So they're very token hungry. Consider the fact that Manus typically calls 50 tools per run. It's a lot of tokens. Anthropic mentions production agents can call hundreds of tools or have hundreds of turns. So it's costly, it's slow, and it can degrade quality. So that's why we talk about agents and agent harnesses. Often, agent harnesses have some mechanisms to manage this. And a few of the trends I've observed are one, context reduction. There's a few interesting tricks here. Some of those intuitive are basically compacting older tool calls. So imagine you've called a tool, then you call another tool. by like your fourth turn you can compact tool call one you don't have to keep it in the message history mandus does this thing it's a good idea i've done it as well another thing if you use clock code is summarization so basically once you start to fill up the context window of the lm produce a summary of the entire message history and that basically compresses all those tokens into much shorter form and you can move forward so context reduction pruning or compacting older tool results and trajectory summarization are two tricks we've seen across cloud code and MANIS. I've used them as well. Imagine if you're using like latest sonnet four or five models with a million tokens. You're saying you don't even want to use those million tokens. You only want to use a fraction of them because of context raw because the agent gets confused if you start to use the whole context length. This is a very subtle good question and it's actually underreported. So it is true that these models have some context window, for example, a million tokens for the latest four or five models. That doesn't mean that performance will be of high quality through the entire context window. And you can have degradation in all sorts of non-obvious ways with respect to context length. The MANAS CSO and I chatted about this last week in our webinar, and he mentioned that often the effective context window for these LMs is actually quite a bit lower than this kind of state of the technical one. So it's something to be very careful of. Just because the context window is a million tokens doesn't mean you're going to get necessarily high quality instruction following throughout that entire kind of context length. And the failure modes can be not obvious and subtle as noted in that Chroma report on context fraud. So I think it's worth being careful, and even if you have a very large context window, being judicious about how it's used. And Anthropics come out and said as much. They have a very nice, actually last week put out a very nice white paper or blog post on context engineering. their new SDK has a bunch of updates that actually incorporate some of these ideas. In particular, it actually has basically a compaction of older tool calls automatically built into some of the new models in the SDK. So they're actually employing this idea exactly. And as noted, Cloud Code indeed uses summarization. For those wanting to dive even a bit deeper, it definitely doesn't help with context, Rob, but there are... tools like prompt caching that can help with cost and latency when having long context as well. Okay, so that's an important point, a good one. So prompt caching is indeed useful. That helps with cost and latency. It doesn't help with context rot. Yes. Because if you're using 100,000 tokens, even if it's cached, it's still 100,000 tokens. So Manus actually uses caching very extensively, but they still perform context reduction through pruning and summarization. So you actually do both. Yeah. The other thing I've seen, and this is anecdotal in work I've done and from people, friends who work in the space is if something's super, super important, a really important piece of information, including it at the start of the prompt and the end of the prompt can seem to increase performance anyway. Yeah, that's actually, that is true. And actually, so I've done that quite a bit. So basically moving instructions to, for example, yeah, the most recent message to reinforce something. And that's part of this. So basically, if you're reducing context in an effective way, the overall message list is managed pretty appropriately. Because what's happening is you're pruning or compacting older messages, particularly the older tool calls, and you're doing summarization. So basically, your message list will be much less token heavy versus if you did not do those things. But there's a related, so there's three big ideas here. One is we just talked about reducing context. The second one is offloading. The third is isolating. I'll talk about those briefly. So offloading actually means, for example, Manus uses the file system. And for all tool results, it's tied to the pruning thing. You actually save the full tool message to a file system. So it's preserved. So then when you do that compaction, you always have reference to the actual file if you ever need it again. That's it. So this idea of offloading is a really good idea. And I use that as well. I think it's a very effective way to do it. The other small point I'll make with respect to offloading is actually offloading a lot of your functions or tools. So this is a very subtle one. We're seeing this more and more. Instead of giving your agent like 100 tools, which can be, it uses a lot of tokens because you have to include all those instructions in your system prompt. Like you have 100 tools, here's how to use them all. Instead, for example, what Manus does, and I've used this quite a bit as well, use only a small number of kind of atomic tools, like less than 20. file system, bash tool, and basically allow your agent to use a computer to, for example, run scripts. That can expand its action space hugely without bloating function calls. For example, this is a good take that Manis mentioned to me most recently, it's top of mind. For all MCP tools, I think we'll talk about MCP in a minute here. Instead of binding all those to the model and having those all live in the system prompt, they just have a CLI that the agent can call through a bash tool to run. any of those MCP tools. So I want to make sure I make this clear. The main idea is instead of binding a huge number of different tools to the model directly, instead bind a small number of tools like a file system tool, a bash tool, and let the model or agent, for example, use a bash tool to execute commands to do many other things. So the action space can be huge, even though it is only making two or three different tool calls. That's the key insights and it's a very good one with cloud code. Think about cloud code. How many different tools is it actually calling? I mean, I've used it huge amount since last February. Search, bash, yeah, web search. I can't think of a huge number more than that. Very simple. That can do a huge amount. Because if you just give it access to a computer, it's extremely powerful. So that's a good insight. Basically, the action space can be expanded hugely if you give agents access to a computer effectively. That's kind of the key insight. Second idea there, and offloading from, like, the system. function calls and the agent instructions to, for example, just calling them directly from, for example, a terminal. The final, I'll mention briefly, is context isolation. This one's a little bit more clear. But basically, when you have a task that, for example, is token heavy, you can also offload it to a subagent, let that subagent perform that task, and then just return some result to the main agent. We see this a lot. I've used it extensively in OpenDeep Research. Anthropic uses this in their multi-agent researcher. Manus uses it. So very common approach. context isolation through multi-agents, very intuitive. So those are the three big ideas, reduce, offload, isolate, with a number of examples from Cloud Code, Manus, and some of my own work on OpenDeep research. Okay, another acronym came up there, the MCP. Langchain has been a really big part of the emerging ecosystem of LLM infrastructure, and now we're seeing protocols like MCP become pretty popular. Can you talk a little bit about how this stuff fits together and what we all should know? Yeah. This is a really good one. There's a really good talk from John Welsh at Anthropic at this year's AI Engineer Summit that talked about the origin of MCP inside Anthropic. It's a good motivating kind of story to how to kind of how to think about this. MCP stands for Model Context Protocol. So as we talked about before, models got really good at tool calling sometime mid to late last year. And when that happened, this is internal at Anthropic, people started writing. all sorts of tools without much coordination. So there's lots of duplication, many custom endpoints for different use cases. And all these inconsistent interfaces basically confuse developers, duplicate functionality, create maintenance challenges, and so forth. So the model of Connish protocol emerged internally as a standard protocol to address this problem. And it was open source and basically is a protocol that allows you to connect tools, context, prompts to different LLM applications, or it's a client server model. So it's basically an MCB server. and a client application. The client, in a tangible case, like for me in my day-to-day work, I have an MCP server that services LangGraph documentation. I work with LangChain. We use LangGraph. LangGraph is our open source kind of agent and workflow framework. I want to write a lot of LangGraph code. I have a little MCP server that connects to the LangGraph docs. That's all it does. But I expose it as an MCP server so I can connect it through the same server to Cloud Code, to Cursor, to Anthropic Cloud Desktop App. So it's like a universal connector protocol to connect tools, context, or prompts to different applications. That's really all it is. Now, the bigger picture here that I think is interesting is relates a little bit more to the broader ecosystem and Langchain frameworks. So one of the points that John made is that standardization is often very beneficial, particularly in large organizations. This gets a little bit to maybe some of your audience, but we've seen this a lot with Langchain and Langgraph. The reason why, certain frameworks like line chain line graph protocols like MCP are popular is because standards are helpful. If you have large work with many different people, a standard set of tooling is very beneficial. And that was one of the reasons why he argued MCP really took hold with Anthropic is because the standard was very useful for a lot of different reasons internally in terms of kind of auth and security and consistent documentation and onboarding and so forth. And we've actually found that's actually one of the main reasons people enjoy, for example, Langraph. Langraph is a framework for building agents, building workflows. It's very popular. It's well-supported. There's good documentation. And so, for example, many organizations building agents onboard to Langraph because it's just a standard set of low-level tools and primitives that you can use to build agents, and everyone's speaking roughly the same language. And it works seamlessly with MCP. So you can build a Langraph agent. You can connect to tools. You can use MCP to connect to it. So that Langraph agent is a little MCP client. You can connect it to MCP servers, no problem. So they play well together. But I think it... The interesting point though is about this notion of standardization and why actually in larger orgs, standards are important. And that's what motivates certain frameworks, protocols for kind of taking hold is my view on it. So now we've been talking around this, but I want to talk about evaluation and in particular, like how people who work a lot on evals almost feel that it should be called something other than evals because it seems niche, right? Whereas a lot of the time it's figuring out if your product works and does what you want it to do. You mentioned that Manus has re-architected five times this year or something along those lines. Yeah, that's right. And if there is a constant need and push for re-engineering as models improve and new models come out, how can you test if your system is future-proof? And how do you think about evaluation and just making sure it works? Yeah, so that's actually maybe two different interesting threads there. So one observation is kind of this idea of you have your system. How do I know it's going to be resistant to newer and improving models. So one of the kind of problems or kind of one of the challenges that can happen is that I have some architecture, it works very well with models today. And so if models get better, my architecture limits further improvement of my application. This can happen for lots of different reasons. And that's one of the major predictions of the better lesson is basically the structure we add to our applications today limits their progress in the future or limits their improvement in the future. You can test this. Basically by, this is a good take that Manis mentioned that I actually really like, you can actually test against different model capacities even today. So take your system, evaluate it against, for example, a low capacity model, a mid capacity, and then state of the art, and make sure performance goes up. If performance goes up across capacities, you can tell that your harness, your system is future proof in that sense. I think that's point one on evals and future proofing your system. But point two, I think is interesting and broader, broadly speaking, talking to Manus, talk to a lot of anthropic people internally as well. A lot of the larger static benchmarks become saturated very quickly. So a lot of the evaluation, for example, Claude Code, they've spoken about this publicly. A lot of it is actually just dogfooding and direct user feedback in-app. Manus mentioned the same thing. A lot of their evals are born from direct in-app user feedback. So getting your products out there, having the ability to capture feedback in app and roll those into eval sets is often what people are doing in practice at Langchain we have Langsmith it's a very feature rich set of evaluation tools we have lots of nice tooling to capture feedback from traces at the eval sets and so there's lots of support for that but I think that's kind of what we're seeing is oftentimes people are attempting to ship product capture user feedback roll that feedback into evaluation sets is kind of the approach that I'm seeing more and more which is obvious I think the only subtle point is oftentimes these kind of larger static benchmarks get saturated very quickly. So you need to be constantly be servicing new failure cases from users rather than relying on kind of these big fixed kind of data sets. Manus said they moved away from, for example, I think it was like Gaia and some of these other big QA benchmarks. They said they saturated relatively quickly on those. And just to be clear, this is amazing because relying on user feedback. is wonderful in these cases but if you're working in a regulated space and serving a conversational agent to financial customers or people coming to an online pharmacy or something like that you can't actually do this yeah that's true may depend on your application so ideally are operating a domain where you actually can get some degree of user feedback to assess the quality of your app i'm not actually sure if that's feasible in all domains but ideally that would be the case yeah actually in your mind It's interesting to think through what good evaluation really requires in this space, because where the outputs are so non-deterministic and context-driven, it sounds like there's kind of these buckets of user feedback, large-scale eval, and maybe there's a bit of interchange between the two. Is it even possible in your mind to build great agents without sprinkling at least of those two pieces? Yeah. For every agent I've built, and for most of the popular production agents, talking to Manus, talk to Claude folks. Evals are certainly being used across the board. They are obviously very useful. I think that the catch is there's been some good blog posts on this. I'll link a few in the show notes. It's always just very important to actually look at your data and not rely strictly on evals. And I think there is, we've seen this quite a bit internally. There's an emphasis on just dogfooding, getting your applications to the hands of users, looking at the raw traces. looking at user feedback directly rather than relying on large static eval sets because the models are changing so fast. And this is one of the interesting things that came up from talking to cloud folks. A lot of cloud code, it was really driven by internal dogfooding. So basically being very aggressive about shipping updates, dogfooding internally, collecting feedback very rapidly, looking at traces and updating it in that manner. That does seem to be a common mode of evaluation. that we're seeing across LN applications. And I think just looking at the raw data, having a good tracing system in place is like table stakes. I think that having high quality evaluation sets is beneficial. And Hamel Hussain, I'll link some of his stuff, has posted a lot of good things on that particular topic. But I do think the case, and Hamel has mentioned this quite a bit, just setting up high quality tracing, looking at your data and being very aggressive about that and shipping, at least dogfooding internally very aggressively is where you start. Absolutely. And I think it's very important, of course, keep an eye on your high level business evaluation are your other metrics you think you want met actually being met but let that guide development and i think a gotcha for a lot of people beginning building this type of software is focusing on the generative part and just thinking about retrieval a lot of the time it isn't the generative part which is the issue it's your retrieval so your evals will guide focusing on on retrieval for example right Yeah, I think a related point there is actually what I found is when you're laying out these systems, it can be very beneficial to set up evals for subcomponents. RAG is retrieval or RAG is a good example. If you have a retrieval system, you're exactly right that the quality of the ultimate output is dependent upon the retrieval itself and then the generation. So you're retrieving from a vector store or database, you're passing it to an LLM, the LLM is producing an output. You can actually do an evaluation on the retrieval itself, and that can be very beneficial. In my little email app example, actually, I have evals for every component of that. I have an eval for just a triage step. I have an eval for the ultimate responses. I have an eval for the tool calls. And so that's actually a good point. When you're laying out these applications, you can have evals for these subcomponents to make sure that these smaller pieces are working as expected. And for your ultimate output, maybe for that, you, of course, maybe have an eval set or two, but you're also doing very aggressive dogfooding or just getting it out there using user feedback to kind of update your data sets aggressively. It is true that in development, typically with applications, you can set up evaluations or subcomponents just to make sure they're working well as you build out your system. You have your whole system, then you ship it. Then you're more a little bit into like kind of guard railing with some like online evals, just making sure that you're not seeing egregiously wrong outputs. So you might be doing that. And then you might also be just collecting user feedback directly, rolling bad examples into datasets more on the fly. Totally. So to wrap up. A lot of our listeners and viewers are AI data leaders, ML leaders. So I'm wondering for leaders trying to make sense of the space, what do you wish more engineering managers or CTOs understood about how Gen AI systems actually get built? Yeah. Yeah. Maybe I'll walk through a few kind of summary principles that hopefully hit a lot we talked about. So I think one, keep things simple. Use just prompt engineering if you can get away with it. If you need a little bit more complexity, then bump up to a workflow. If you actually can't get there with a workflow, then you can consider an agent. If the problem is truly more open-ended. An agent, though, with try to minimize the tools, keep it very simple. If just a single agent can't get you there, then think about contact engineering. You could, for example, offload to multi-agent through if you need a contact isolation for more like heavy-duty tasks. And finally, if all those things are insufficient, you might think about fine-tuning or training models, but really only after all the others are exhausted. So I think that's point one, keep things simple. And I think it's tricky because you often might hear, for example, on Twitter or in the timeline, people talking about, oh, reinforcement fine tuning or building agents. And maybe your problem does not need that. Don't increase the complexity arbitrarily. The second point is like the bitter lesson thing is building for rapid model improvements. So recognize what you build today will have to be re-architected kind of aggressively over time. The Manus example, five times since March. My example of OBD research, I re-architected that three or four times in a year. You have to bake in the fact the models are getting much better and whatever little kind of crutches you have in your application today to make it work, go away as that model improves. That's exactly what is predicted by the better lesson. And you have to be aggressive about removing those kind of assumptions or structure as the models get better. Otherwise, you might bottleneck your performance. So that's a very important thing to think about and keep in mind. And relate to that, don't be afraid to rebuild. Manus rebuilt five times, Cloud Code's constantly rebuilding. I rebuilt OpenDB research three or four times in a year. So you have to embrace that in this new era of LLMs. And I also think a subtlety is that the cost of rebuilding is much lower with code models. So it's much faster to re-architect things. Another subtle point I'll make is things that don't work today will work tomorrow. So Cursor's a great example. This Cursor did not work well until Cloud 3.5 saw it. And then suddenly the product experience was unlocked and it... Obviously, the rest is history. So actually, don't be shy or afraid or whatnot if your product doesn't quite work yet. Because, for example, of model efficiency, that'll very quickly get removed as models get better. I think Kursh is a great example of that. Yeah, and maybe the final point is just be wary of rushing to train models. It can be tempting. It can be really charismatic. Think about applying fine-tuning for your domain. But oftentimes, these... The frontier model is getting so good so quickly. You can take all this time to collect a dataset, train a model, and then actually you get bitter lesson because the frontier model kind of encapsulates the capability that you fine tune for. And then you waste all that time. I'll give you an example of that very specifically. Two years ago, structured outputs weren't great from frontier models and people are doing fine tuning for structured outputs. And yeah, it's all relevant today for the vast majority of use cases is irrelevant because. the LLM providers have gotten extremely good at structured outputs and complex nested schemas and JSON mode and so forth. So just an example. So keep it simple. Build for rapid model improvement. Don't be scared to rebuild. Things that work today won't work tomorrow. Don't rush to train models. I think are like the five things I would leave you with. Fantastic. What wonderful lessons, both from all your time building, but everything you've seen happen in the space working online chain as well. Thank you for bringing us your wisdom and expertise and for your time as well, Lance. This has been super fun. Great to be here. Great to see Duncan again. And great to meet you, Hugo. Hopefully I make it up to Sydney sometime and we can get in the water. Go for a surf together. Yeah. Thanks so much for listening to High Signal brought to you by Delphina. If you enjoyed this episode, don't forget to sign up for our newsletter, follow us on YouTube and share the podcast with your friends and colleagues. Like and subscribe on YouTube and give us five stars and a review on iTunes and Spotify. This will help us bring you more of the conversations you love. All the links are in the show notes. We'll catch you next time.

Lance Martin of LangChain joins High Signal to outline a new playbook for engineering in the AI era, where the ground is constantly shifting under the feet of builders. He explains how the exponential improvement of foundation models is forcing a complete rethink of how software is built, revealing why top products from Claude Code to Manus are in a constant state of re-architecture simply to keep up.


We dig into why the old rules of ML engineering no longer apply, and how Rich Sutton's "bitter lesson" dictates that simple, adaptable systems are the only ones that will survive. The conversation provides a clear framework for leaders on the critical new disciplines of context engineering to manage cost and reliability, the architectural power of the "agent harness" to expand capabilities without adding complexity, and why the most effective evaluation of these new systems is shifting away from static benchmarks and towards a dynamic model of in-app user feedback.

00:00 Introduction to the Democratization Shift in AI
00:39 Key Shifts in the Generative AI Era
03:00 From Traditional ML to Generative AI
05:37 Core Principles in Modern AI Systems
07:58 Building and Evaluating AI Agents
08:58 The Bitter Lesson and System Design
14:53 Understanding AI Workflows and Agents
23:49 Ambient Agents and Their Applications
26:22 Building Smarter Autonomous Agents
26:58 The Importance of Context Engineering
28:02 Managing Token Usage in Agents
29:09 Strategies for Context Reduction
33:03 Offloading and Isolating Tasks
36:06 Introduction to MCP and Standardization
39:25 Evaluating and Future-Proofing AI Systems
46:32 Key Principles for AI and ML Leaders
49:52 Conclusion and Final Thoughts