Stop Prompting, Start Engineering: The "Context as Code" Shift

We have effectively had general-purpose agentic development machines for 50 years. We just called them software engineers. You never just wrote somebody a Slack message and expected them to go build an entire system and just make all the best decisions, right? Well, nice to meet everybody. My name is Drew Knox. I'm the head of product and design here at TESOL. I'm going to talk today about... using skills as like in a more professional rigorous like software engineering mindset right so before I get into this why should you trust me well one maybe don't maybe be skeptical but in my past life before leading product and design here at TESOL I was a research scientist leading the language modeling teams at Grammarly and at a startup that sadly has not found success yet called Cantina. It's a social network, AI first. And I've done a lot of work on developer tools, and so I do a lot of moonlighting actually writing code. Probably none of it as good as the actual people on Tesla's teams. I've thought a lot about this. I've done a lot of work on this. I'd like to share some insights. Would love questions. Would love to hear folks like your experience, what's worked for you. So I'll try to save lots of time for questions at the end. But without further ado, you want to work on skills. Maybe more broadly, you want to work on context for your agents. So I'm sure folks have heard about context engineering. It feels like every year we're told that this is the year of something. I've heard people say that this is the year of context engineering. maybe it is, maybe it isn't. You want to get into context engineering. Some of you may have already gotten into context engineering. As you start to work on this, you'll probably go through the same sort of stages of denial, acceptance, etc. of this is amazing, I'm getting good results, to suddenly, God, how does any of this work? Is any of this impactful? I thought I was an engineer, now I feel like I'm an artist or a librarian. How do I turn this? thing, agents, context engineering back into the kind of reliable, predictable engineering that I know and love. So how do we go about doing that? I think the first thing to realize, just in case anyone has not sort of conceptualized this, the reason we're all doing context engineering now is because we've all effectively become tech leads instead of ICs, right? Like the job in some sense is no longer writing good code. It's ensuring that good code can be written. which is things that tech leads know and love or hate already around you want to be maintaining good standards, making good decisions, documenting it, providing the context to the rest of your team, setting a good quality bar for other engineers to contribute. We're doing that. We're just doing that for agents now. And so what that means is that... context is in some sense our new code some people might hate that i say that so please take it with a grain of salt it's a metaphor uh if context is our new code though there's things that we expect of our code right we want a way to know are my programs correct are they performant how do i reuse programs how do i automate repetitive tasks that are annoying and so we've we've come to expect a lot of answers here for code, like actual code that we write. I won't belabor the point, but we have all these things, right? Unit tests, integration tests, we have analytics and observability, all these things that give us really good insight into how our programs function. And the core thing that I wanna argue today is that all of these have an analog in the world of context engineering, and if you are sort of diligent about finding a tool set that does this, I'm gonna show you all TESL just to illustrate the concepts, but you don't have to use TESL to do any of this, these are general concepts and patterns, you can reclaim a lot of that predictability, a lot of that sort of rigor that you've come to expect with code. So how can we take all these concepts and apply them to context? That's the sort of TLDR. Before we get started, there's three challenges that make it not a direct comparison, right? I'm also, as I look at this, just realizing the image that was generated here made a fuzzy correct symbol. I was looking at it through a fuzzy display, so I'm now appreciating my own slide a bit better. So we have three problems. The first is that LMs are non-deterministic, right? You can't just run them once and say, oh, it worked, or oh, it didn't work before I now know my context is good, right? If you tell an agent to do a thing, Sometimes it will, sometimes it won't. I'm sure I've felt this pain many times. The second is that a lot of times when you create context, there's not one right or wrong answer. If you write a style guide or if you write a documentation for a library, how do you determine that an agent's solution did it correctly? You can't just write a unit test and say, ah, we're done, it worked. So grading output can be a little challenging. And finally, there's this new problem that your programs are now actually things that describe other things. So you have things in sync. So you might update your API, you need to update documentation to match it. Or you might change a company workflow in one place to make sure that it gets distributed out everywhere across your organization. So sort of a new problem falls into a lot of similar automation goals that you might have. might have. So this is like a quick overview slide. I'm going to dive into each of these. What is the analogy? And maybe I'm trying to get a little too cute with this, but I actually do think there is a direct analogy for all of the tools that you've come to expect in the software development lifecycle. So I'll quickly run through them. The first is that we have things like static analysis in Codeland. That is going to look a little bit more like an LLM as Judge, but the same idea of You want to have a fixed set of best practices, of rules, of validation, of compiling, all those things you should be able to run against your context. To give an example of something we saw recently from a customer using TESL, they had added an at sign into one of their files and didn't realize that that was suddenly triggering the import mechanisms for most agents.md. and it was just breaking a whole host of their context without even realizing it. So it seems silly, but static validation still important with your context, right? So you want to have that set up. You want to have linting. You want to have CI, CD for it. Unit tests are going to look probably the most different. Now, instead of defining a unit test that runs, you're going to want to think through scenarios that sort of stress test the agent. Does it do what you want? and then run it many times in parallel, and you're going to take statistical averages. And so you want to see when I add context, does it actually improve the sort of average performance of something? We'll get into that. That's probably the hardest one. That's the one where it feels a little different. It's not going to be quite the same, but you can still get pretty reliable. Like what I did has improved. Integration tests, same thing, but testing lots of context at once. So you're designing scenarios that map to using different kinds of context together. And then we'll get into some of the more esoteric things that Tesla doesn't even fully support yet, but we sort of see on our roadmap and we think everybody should be thinking about it. Analytics. So how can you start actually measuring agent sessions in the wild to see what's happening? Do we have missing context? Are things being used correctly, et cetera? And then also getting into automation, like build scripts, how do you make it so that your context is not this static thing that grows out of date and dies, but as you update things, you're getting follow-up PRs that auto-update your context. And then package manager reuse. This is actually kind of in the last two, three weeks sort of blown up everywhere, things like SkillsSH, things like TESL's context registry. The idea of reusable units of context has sort of come onto the scene, so I'll talk a little bit about unique problems with how you pick what sort of context manager you're gonna use, or package manager. I guess both work. Okay, so review formatting and best practices. Like I said, I'm gonna use TESL as an example here, but I'll try to explain all of this in a very basic, like, you could build all this yourself if you wanted. There's other tools that do a lot of this. Not as well as TESL, though, obviously. So this is pretty easy. If you look at things like the skills standard, so I'm going to use skills as an example here. If you look at the skills standard, there, first of all, is a bunch of just static formatting you can do. I believe they have a reference CLI implementation that will just verify your skill compiles. I think everybody who's writing skills should have that in CI CD, checking that all of your skills are kept up to date. Like anytime a skill file changes, you should be checking validation. You would be stunned how many people, like none of their context is loading and they don't even realize it. So that's a big one. But also if you look at Anthropic, they have a best practices guide. And it's basically a list of things like, so here I have validation. TESL will tell you if your things compile. We also will take just Anthropic's best practices. And we just run that through lmsjudge. There's a little bit more you can do to tune the prompt to get better results, but honestly, just put a prompt that has Anthropix best practices in it is a great starting point. And get information on how specific is your context, does it have a good concrete case for when it should be used? I'm sure folks have heard about skills and how they don't activate very often. There are concrete things you can do without even running the skill to know how likely is it to trigger. So these things are cheap, they're quick, you can put them in CI CD and it's a surprisingly large lift to actually making your context useful to you. Recommend like this is just kind of like table stakes. Everybody should have this just like everybody should have a format or a linter or things like that. Bonus points, you can feed the output of this back into an agent and ask it to fix it. It's a pretty nice quick loop. Okay, now slightly more complicated, slightly more of like a net new concept, right? So how do you write evals for your context? So depending on if you're coming from more of a software background or more of like an ML deep learning background, this might either be obvious or not so obvious. But the simple thing that you can do is, or sorry, actually, before I get into this, the thing you're trying to answer is, is my context actually helping? how well is the agent doing at the task that I am trying to achieve? So if I use this as an example, we have some library that we want the agent to use, and we can see how does it perform without any context. We see things like it is not good at using the list function. Maybe it implements it itself, maybe it uses a different library or something like that. It's also bad at async handling, but it's pretty good at correct stream combination. It's pretty good at doing zip files, I'll explain how we measure this in a minute. You wanna understand this so that you can then understand where do I need to apply context to fix the problem. So there's a couple things you might get from a view like this. The first is you might have written a bunch of context only to realize agent did fine without it. Like why are you wasting tokens on it? Or you might actually write something and realize it made performance worse because something's gone out of date or it's just added tokens for no reason. In an ideal world, you see like, ah, it works better with it, and I've only applied tokens where it matters. So you want to be able to answer questions like that. I have here on the right side, you'll notice that we have like a file viewer here. All you have to do to get this set up is write some prompts, right? Like write some realistic tasks that you want the agent to do that requires usage of the context that you've created. and then write a little scoring rubric for what a good solution to that problem looks like. The reason I say write a good scoring rubric and not something like write a bunch of unit tests is twofold. The first is that unit tests are really obnoxious to write and they take a long time and you will quickly find that you just don't do it if you have to create example projects and test suites for every single piece of context that you want to verify. More importantly though, I'm sure many of you have seen that Agents do unspeakable things to get unit tests to pass. And so functional correctness is not the only thing that you're measuring, especially for context. A lot of times you want to know, was idiomatic code written? Or did it use the library I actually wanted it to use instead of implementing its own solution? And there's really no way to measure this with unit tests. It's much better to do more agentic review or LLMS judge. So what you're gonna wanna do, I see somebody taking a photo, so I'll let them take the photo. What you wanna do is define, we put them in markdown files, so Tesla will generate these and give them to you. You could write them yourself. You wanna have a prompt that runs through, like, build this thing, here are the requirements. It should require using the context, or at least should require doing what the context says, right? Because you actually wanna measure it with and without context to see if the agent is just smart enough to do it. on its own and then importantly you want to define some kind of grading rubric so here this is if you were to specify it for tessle you'll say what does a good solution look like and you want to be pretty specific here so you get reliable results from an lm so you'll say things like the solution should use this exact api call somewhere in the in the method or it should implement or it should initialize this before it initializes that right like very granular things that can then be checked at the end of the solution to see was good code written. An important thing to note is that once you have these in place, like this can take a bit of upfront work, it's sort of like the new source files that you have to care about as an agentic developer, but say you get about five of these per piece of context, that's what we found is like a pretty reliable measure. Once you have some of these, then you can kind of reap the benefits of it forever, just like unit tests, right? You can just, every time you make a change, you rerun these, you see if it helped, if it hurt. One thing that's different is oftentimes you'll rerun these without changing the context because there is something else that's changing, which is the agent and the model. And so a lot of times you'll rerun. What we have found is oftentimes you can start stripping out your context as agents get better. Like we had style guides for Python. Claude Opus 4.6 writes pretty damn good Python. It doesn't need a style guide anymore. Your evals can tell you that and help you delete context that you no longer need, save money, don't pay the tokens. Every once in a while there will be a regression. There was, I think it was Gemini, a recent Gemini was kind of a smart ass and thought it didn't need to use tools and read context. And then we realized, oh, we've had a regression, we need to go beef up how much we tell the agent to use the context. Okay, so repo evals, I talked about integration tests. I won't... go too deep into this because it's basically the same thing, but you don't want to just test your context in isolation. That can help you like debug, are you saying the right things? Have you, has your context grown out of date? You also want to measure realistic scenarios in your full coding environment, all your context installed. I was just watching a talk earlier today that described the dumb zone, which is where you've gotten too much context in your context window. because of tools, because of context, because of all these things where agents are just persistently bad. So you want to have a few coding scenarios, again, like five for your repo is a fine place to start, that just represent an average development task. And again, same, like a rubric to grade the output and just run it every once in a while. See if your tech debt has gotten to a point where agents don't understand how to work in your code, have you installed too much context, have you installed too many tools, all of that. This is something that Tesla is building as well. One thing we found that works pretty well is like scan your previous commits and turn some of those into tasks. And you can even have like on a regular cadence, pick like five random commits over the last month and just refresh your eval suite. For folks who are in the ML environment, you have things like input drift where you want to update your tasks every once in a while. Don't worry about it if that is like seems like too much effort. Just start with something and you can improve it over time. But yeah, that's a, same idea, task scenarios, grading rubrics, run them every once in a while, make sure you haven't degraded things. Okay, so this one I think is pretty cool, but is also kind of scary, which is you want something like analytics and observability, right? You've written this context, you've validated the change before you've pushed it out to the repo for everyone. we do that in software but then we also still have crash logs we have metrics we have usability funnels things like that this actually does exist for agents just a lot of people aren't paying attention to it all of the agents store all of their chat logs in files in accessible places and so you can write your own scripts if you'd like tesla has capability to gather these uh opt-in of course because obviously it's very sensitive information You can review those transcripts to see things like were tools called or how often was this piece of context used? How often does this pattern get actually manifest in the code? How often does it import a library right in the middle of a function? And so there's a lot of rich information here that you could just write a quick script, ask everyone on your team to run it once and just aggregate a bunch of logs and just review common problems that you might want to make new context for. Great one is anytime the agent apologizes. So you can just look for the word sorry, look for the words you're absolutely right. All of these things are good, like, oh, maybe we should write something to fix that. There's a wealth of information, and I guarantee you've got like three or four months of cursor logs sitting on all your devs team, devs machines that you could mine for like, what should we be doing different? This is my warning. I've shifted the color because the next screen is in light mode. It's a GitHub page, so please prepare your eyes. How do you keep your contacts up to date? See, I tried my best. I really tried to hit the midpoint between black and white, so sorry. There is, again, you can do something pretty simple here where you can just set up, there's all kinds of agentic code review tools, there's Cloud Code Web, but I think a general thing to set up in your CI CD if you have created context is any time a PR comes up, have something scan that PR and then look and say, is there any markdown file here that should be updated? So it's not that hard. It really works better than you'd think. It seems like a really complicated task, but because PRs tend to be so focused, agents are pretty good at finding out where they should update. If your PRs are too big, maybe it's a good sign to make your PRs smaller. Again, very simple. So like Tesla can automate a lot of this where it says, oh, you added a new case to your logging levels here. Update your documentation as well. This one is probably the most important. because as your context gets out of date, it just destroys agent performance. So if you're gonna write context, you have to have a solution for keeping it up to date. Agents are pretty good at doing this, so you don't have to do it by hand. Don't do it by hand, because you won't do it. Okay, last thing, package managers. I mentioned that this one's kind of normal. You need a package manager if you want to reuse context, right? code review skill, a documentation on how to use React, best practices, et cetera. So I won't belabor this point. There's lots of good options out there. SkillsSH is probably the most popular, though it pains me to say that. Tesla has a package manager as well. It's not the most popular. I think it's the best. I won't pitch you on why it's the best, but it's the best. The important thing, though, I'm showing here, Tesla's package manager. Ooh, look, we give you evals. We give you all this useful information. two things that are different that you should just think about when figuring out how to use context. The first is that unlike other package managers, a lot of context that you're going to install is going to be describing other package managers, right? So I have an example here where I have documentation on a library that's part of PyPy and it describes a particular package at a particular version and these are sort of like a net new, maybe not entirely new, but it's a weird concept. And so you just want to think about what is your strategy for matching, if you have documentation on a library, how do you make sure that as you update your library, you keep documentation keyed to the same version, right? You don't want to say, I'm using context seven on the latest of React, but actually you're pinned to like React 17 or something for some reason. There's lots you can do here. You can just write, again, something yourself. A lot of this is not necessarily hard to do. It's just fiddly to keep updated and keep sort of pace with the rate of agent change. But think about how do you keep your context in sync with dependencies, in sync with tools or APIs that you're using because it's a new source of drift that you might have to care about. So that's it. That's my walkthrough. I'm happy to answer questions now or afterwards. I think I have a few minutes, so I will turn it to questions now. I should have had a question slide, but instead you're going to look at this. That's all right. Thank you very much, Drew. Yeah, we have a few minutes for questions. Please wait for the mic if you have a question. If you do have an excellent question, then we have some merch that Sam is modeling over there, either a bag or a fidget spinner or a hat. Drew takes questions on TESOL, but also Fast and the Furious and Improv Comedy. First, this is Drew. First question. Thanks. Hi. Great presentation. So what do you see as the end state in 12 months or even six months if Claude 4.6 is just really good and Codex 5.3 and then when Codex 6 comes out and Claude 5 comes in, Gemini 4. I mean, do we need a lot of the scaffolding or does it go away? Fantastic question. I would say, first, it's going to split a lot by whether or not you're a green field or a brown field. Like if you've built an app from the ground up. for agents, it's going to be a lot easier than if you're doing like an enterprise Java app. So that's one. I think the number of things you need context for will go down. So the example I gave, Python style guide was like all the rage six months ago. Nobody needs it now. But describing your custom internal logging solution, you're always going to have to document that because an agent doesn't have access to it, right? It's not in its training weights. So there's some amount of knowledge that will always need to be told to the agent. My expectation is that eventually you won't be proactively jamming almost any context into an agent's window. You'll have some kind of signposting like progressive disclosure. The agent will get to look at it. if it deems it necessary, like a normal developer will. And then a lot of your usage of context will be applied at review time. So you will create a review agent that looks for things like, did it break our style guide? Did it re-implement something? And it'll just be there for control. It won't be there to sort of like educate the agent up front. I think evals are gonna play a big part in helping you navigate that change, right? Knowing when it's time to move things out of the context window into a review or just delete it. Awesome, thank you. Next question over here. Hi, I wanted to ask about evals. You had max score 50 and 30. From my experience, non-binary score doesn't really work. Could you tell how it works and for what agents does it work? No, I think that's right. I think binary is pretty much the only, like, we give granularity in TESL if you want to do more, but if you look, agents pretty much always score zero or max score. So I would say no. You could get away with zero or one and be about the same. So I'm an AI engineer, so I want to build solutions really fast. Would you recommend just using Opus 4.6 to get out an eval set very quickly and then just use that as a baseline, which is not perfect, but just have that as a starting point? Or would you recommend not do a full thorough analysis, get a baseline, and then continue? So let me make sure I got that right. But like, should you try really hard to optimize what agent you're using and the exact, yeah. Personally, I'm busy. I have a lot to do. Just start with the best agent. And then I think what I'd say more is once you have some really repetitive tasks, it can be worth it to say, okay, what is like the best, the cheapest I can get away with? A lot of times context will help you use smaller models to do that. But no, I would say for a day-to-day, like your general driver, just always crank it to the max, unless you have some reason you can't do that. I think this might have to be the final question because we have another talk. Yeah, my question is more towards probably like a non-technical or people are not too technical. When do you think or what barometer do you think we can use to measure the point in time where we don't really need to have two? into what the agents do or what the LLMs do, for example, like you guys have what, spec-driven design or acting as like a product manager, so we say you're a PRD, but you don't have too much input, or are we gonna be at that soon? I'm gonna throw out maybe a spicy take, which is that definitely we're not there, and I don't think we'll ever be there. What I mean by that is we have effectively had general purpose agentic development machines for 50 years. We just called them software engineers. And in that case, you never just wrote somebody a Slack message and expected them to go build an entire system completely unsupervised and just make all the best decisions, right? Another way of putting this, my wife, who is a very senior staff engineer at Meta, is if you cloned me, I would still code review my code, right? Like I would never accept anyone to submit things without looking at it. So personally, I think there will always be a place for a technical architect, a steward, somebody who's guiding the sort of quality of the code base. I think what that role is will change over time. So right now it's a lot of like in the weeds, very specific decisions. It's a lot of like reviewing code. It's mentoring and coaching people up. and you tend to have like 1 p.m. to 5 to 10 engineers, right? I imagine we'll get to a place where you invert that ratio, and you have one sort of like technical steward whose job is to think about the overall system design, to be constantly reviewing agent code, to be reviewing things that people are building and understanding like, oh, this is a consistent failure point. If we abstract this part out, if we build a component that agents can use. They'll more reliably get better, like one shot success. And then you have like five to 10 more like product design, product engineering people who are out sort of exploring the frontier of your product space and then have this one technical steward helping them land their code and keep things maintainable and improve, like improving them over time. When will we get to that point? That part I'm less certain of. It could be in two weeks, it could be in two years. I think it's probably in the order of single digit years, certainly. I wouldn't be surprised if it was a completely AI native Greenfield project starting within the next year could work in that model. But certainly for Brownfield, I think it'll be harder.

In this session, Dru Knox (Head of Product at Tessl and former Research Scientist at Grammarly) moves past the "magic" of AI agents to discuss a more professional, rigorous software engineering mindset for context engineering.

As we shift from individual contributors to "tech leads" for our AI agents, the quality of our code is increasingly determined by the quality of the context we provide. This talk explores how to reclaim predictability in agentic workflows by applying classic development lifecycles—like static analysis, unit testing, and CI/CD—directly to the context layer.

What you’ll learn:

• Context as Code: Why context is the new "source code" and how to manage it with the same expectations for correctness and performance.

• Engineering through Non-Determinism: Strategies for grading agent output when there isn't a single "right" answer and how to handle the inherent variance of LLMs.

• The Context Lifecycle: Mapping traditional dev tools to the agent world:

    ◦ Static Analysis: Using LLM judges to enforce best practices and validation rules.

    ◦ Unit & Integration Testing: Stress-testing agents through parallel scenarios and statistical averages to measure performance improvements.

    ◦ Observability & Analytics: Measuring agent sessions in the wild to identify missing context and usage patterns.

• Automation & Reuse: Moving away from "static context" that goes out of date toward auto-updating context and reusable context registries.

About the Speaker:
Dru Knox is the Head of Product and Design at Tessl. Previously, he led language modeling teams at Grammarly and has spent years building and moonlighting with developer tools.+1

#AI #SoftwareEngineering #AIAgents #ContextEngineering #LLM #DevTools #Tessl