Build Hour: API & Codex

Hey, everyone. Welcome back to OpenAI Build Hours. I'm Christine. I'm on the Startup Marketing team, and today I'm joined with Charlie and Ryan. Hey, folks. How's it going? Hey, everybody. Awesome. So Charlie is on our DevX team, and he will be leading the session. And Ryan actually came all the way from Seattle to be with us live in the studio today. And he's going to be chatting about the future of work. So today's session is all about API and Codex. And if this is your first build hour, the goal of the session is to empower you with the best practices, tools, AI expertise to scale your company using OpenAI APIs and models. And this session in particular is really going to teach you how to use Codex for all of your engineering work and really what is the next step in using Codex that goes beyond pair programming. So here's a snapshot of what you can expect today. We're going to spend five minutes on all the new features that we launched across Codex and the API. We have lots to share on the app as well as a new model that just came out last week. So perfect timing. Then we're going to dive into a demo on agent legibility score. So more to come. And this is going to be super practical for you to apply right away to your team. And then Ryan is going to talk about harness engineering. So we released a blog on harness engineering from Ryan, who's going to be actually in person talking to you about what this means and how you can apply this to your team. And then we'll save some time for a customer spotlight basis. We'll be joining us live. And we're really excited to have the co-founder, Mitch, to actually talk about how he uses Codex with his engineering team. And then last but not least, we'll save 15 minutes for Q&A. So on the right side of your screen, there's a Q&A chat box and you can submit your questions. We'll answer them during the session and then save a few to answer live at the end. So with that, I'll turn it over to you both. Thanks, Christine. So let's kick it off by talking about what's new in Codex. You know, we've really entered this third phase of using AI for software development. The first phase was, you know, autocomplete, right? Sort of the ghost text. The second phase was pair programming, where you would have a model sit alongside you in your IDE and work on generating code, potentially merging it into your existing code. But we've really come into this new era of agentic delegation, and that's where you can use things like the Codex desktop app to manage multiple agents across increasingly bigger and bigger tasks and workflows. I'm sure you all have felt this, but there was a huge step change in capability with GPT 5.2. and with gpt 5.3 and gpt 5.4 on top of that we have only trended increasingly into this long horizon full delegation to the agent direction and for me as a builder this has been a fantastic thing to empower me to do more and to do more quicker absolutely i think it's been kind of surprising for me in terms of how quickly my own workflow has changed especially with the codex desktop app and you know it's mostly replaced the ide for me at this point And so, yeah, we've had, you know, just in the last couple of months, like you were mentioning, 5.3 Codex, which got even faster. The Codex app, which is now about managing agents rather than necessarily living in your terminal or your IDE. And we just released GPT 5.4, which we'll get into in a little bit. We'll be using the Codex app a lot today. And if you were not aware, it's now available on Windows, which is a big milestone for us. Super excited about this app, and one thing that is truly amazing about it is has Windows native sandboxing through and through. It doesn't rely on WSL and allows you to build with Codex in ways that are native to your local development best practices. I think it's super cool that we're bringing all the power of Codex's security model and best-in-class coding capability to developers on Windows. We really aim to meet you all where you are to help you build. Yeah, so much of the world is building on Windows, and so we'd really love it for you to check out the latest Codex app. The app itself has a few new features in the recent weeks and months. Big ones are skills and apps. So you can bring capabilities and expertise via skills. There are a number of them that are recommended and come pre-available inside the app. And then we also have apps which you can share across both ChatGPT and Codex. And you can use those. They were previously known as connectors, but you can use those to connect ChatGPT and Codex to the tools that you use every day. Yeah, skills are super powerful. The way I think about them is context that we give to the agent to show them to what end do you use the tools you have and how do you use your tools well. And the fact that we're bundling that bunch of them with the Codex app means you get all the best parts of how we use our tools well to supercharge your own workflows. And let's talk about what's new in the API. So the big news from last week is GPT 5.4. It's got native state-of-the-art computer use capabilities. You'll see us refer to this as CUA, computer use agent. It supports up to a million tokens of context, and it has a new tool search tool. We found that in practice, a lot of the cutting edge, the bleeding edge builders were using hundreds and hundreds of tools, which was potentially... creating issues with context management. So we now have tool search, which can help you namespace those tools. And the agent can intelligently sort of, you know, progressively expose those to get the tools that it needs. Yeah, the term of art here, if you all have heard it, is progressive disclosure. The idea that not all the tools need to be in context all the time for every task the agent does. So instead of giving them everything up front, we hide all the tool descriptions and individual calls behind something that allows the agent to do a little bit more natural discovery and choose which parts it pulls into context intelligently. Exactly. And it's the most token efficient reasoning model you can see in our benchmarks. It's achieving the same performance as 5.3 codex and 5.2, but with a fraction of the token consumption and latency. We're not going to touch on all these today. I would encourage everybody to check out developers.openai.com to look through the API change log. But some of the big things we've shipped just in the last month alone, the Kua API, which I mentioned, that really beefs up the ability for developers to manage browsers and computers. We have a new mode, a code mode, where you can actually run. The model can generate JavaScript for you to run in a REPL. And so that dramatically speeds up the time it takes to do. Previously, you would have to sort of click and screenshot and find coordinates, and now you can compress that all into like a single JavaScript execution statement. Love to give the model more tools and let it cook. Yeah. We have skills in Hosted Shell, which we will be using in this demo. Skills, you can upload to create a skill ID, which the model can reference. Hosted Shell gives the model a container environment, which it can spin up, and that can actually execute bash commands inside of that container environment. And then we also have WebSocket mode, where if you're building an integration on top of the API, you can switch over to using WebSockets. And for extremely tool-heavy use cases, we find that it improves latency by 20% to 30%. One of the things in here that I'm most excited about is the hosted shell tool, because it takes all the magic of coding agents, of giving them a computer and a shell in order to go cook and solve really complicated problems, and puts it in our API. in a way that is super customizable able to be embedded deeply into the agents that you build and optimized to the way you eval the workflows that you're building toward and if you want to use the codex harness itself we do make that easy too by shipping adapters with the agents sdk to allow you to give codex to your agents as well So we kind of give you the best of both worlds here with both of these options. 100%. Yeah, our vision is to meet developers where they are and give them primitives, whether it's hosted on our infrastructure or running locally on yours. Cool. So with that in mind, let's jump into our demo. one of the principles that we're going to talk about you know in the harness engineering portion of today's build hour is trying to make repositories more legible for agents right and so we had this idea of using codecs to actually make an app that will score a get a repository along a certain set of metrics that we've predefined right so this is exciting because it's an easy checklist that we're going to put together that will guide you in the direction of trying to remove humans from the loop from different parts of the code authoring process yeah so i'm here in the codex app i've got a new folder here in the interest of time i've actually created a plan in advance for the agent to go ahead and implement this is the kind of thing that you could easily create yourself with plan mode in the codex app but we've got you know about 10 minutes here so now i'm going to go ahead and tell it implement plan.md right and so behind the scenes The model is going to, you know, think for a minute, decide to explore the workspace and figure out, you know, what files are available and what the current state of the GitHub repository is. And it's going to take a look at what it can do. This is super exciting here because the app makes this much more digestible for me. It kind of lifts me out of my terminal and puts me into a higher level operating mode where I'm more. reviewing the agent rather than being deep in the weeds with it. And this is sort of the transformation and how we interact with this tool that has been enabled by this huge step change in capability that really only became possible this year. There's something interesting here. The app is actually, the model has actually realized we built a previous version of this app in an archived folder, and it's sort of going in to take a look at that code to, I think, learn from previous runs. this is a kind of a peek at an interesting technique that i have used to build some agents for myself which is to basically provide pointers to previous trajectories previous iterations of the task and this kind of serves to give the model a reference for how the task was completed previously what it might learn from those previous trajectories and kind of gives you a very cheap way to implement what we call episodic memory a way for the agent to improve the way it works over time while completing similar tasks. While it's working, let's take a look at some of the things going on in the Codex app here. At the bottom, we've got a task list. This is not anything that we generated or specified. This is just the model internally saying, here is the roadmap that I need to complete and giving us some visual indicators on what it's doing. This is distinct from plan mode. It's really just a nice way for me to have a high level site of what the agent thinks is up next for it. And I use this as a tool for myself to figure out whether or not I should interrupt or provide steering to Codex as opposed to just kind of letting it cook. Helps me have confidence that the agent is doing the things that I expect. Yeah. And so if we open up a new thread, yeah, we can see if you want to toggle on plan mode, it's really powerful for. getting contacts, asking you what direction you want to take things in, and developing a really, really thorough plan. We now have speed, and you can choose whether to run Codex in standard or fast mode. Fast mode is fantastic, but it does consume your usage a bit more quickly. And then we have here, I think, the ability to toggle some of the permissions and settings. I've been using local here, but I love the WorkTree capability of the Codex app. So WorkTrees, if you're not familiar, are essentially like Git sandboxes. And they allow you to work on multiple things simultaneously within the same folder without stepping on or clobbering changes in a different task thread. If you've seen folks with five screens and 18 Tmux tabs, WorkTrees is what they're using in order to go in parallel here. And I think getting your code base working with WorkTrees is one of the steps you can do to move toward a harness engineering mentality. uh basically make it really cheap to work on multiple copies of your code at once yeah and the codex app has a really great handoff feature where if you're in a work tree and you do want to bring those changes back to your local branch you can click a single button to do so or if you want to push them back to the work tree you can click a single button to do so There's a lot of great Git support in here. This is a pretty simple repo, so we just have the main branch. But you can easily commit and push just from the app. You can also open up these files in your IDE of choice. And then there's a few built in. There's a terminal, so we can take a look at what's going on in here. There's a div panel. I think in this case, we've got quite a lot of changes now. It's written 8,200 lines of code just in the few minutes we've been talking. Over on the left, we've got the skills and apps that we were talking about earlier. Really love, I think, just the set of skills that we provide out of the box here, as well as, I think, the apps that you can go ahead and install up for yourself. One neat thing here with Google Calendar in apps is it kind of should maybe help get you thinking about ways you can use Codex to automate other parts of your work other than just the code that you're writing. I've vibed up a couple of tools to help me digest my calendar and make sure I'm allocating my time effectively. Really cool to unblock the other parts of my job with this tool. I love that you brought that up. So let's talk about automations, which to me is like the killer feature for the Codex app. Automations, and you can kind of see a lot of my messy inbox here, is a way for you to just run a command on a regular schedule. And it sounds simple, but I've kind of been surprised at how powerful it is in practice. Yes, yes. One I run all the time is to have the Codex app review all my open PRs and make sure that they're mergeable, that they haven't come into conflict with me. Codex is really good at resolving merge conflicts, and I am super lazy. So I am more than happy to delegate this work to Codex to make sure that I'm always unblocked to smash the merge button as soon as my code is approved. Yeah. In my case, I think my most popular one is Slack management. So just checking on work streams that I've got going on or updating to-dos from Slack. I've seen other folks at OpenAI use it for things like, hey, take my most recent PRs and just review them for any potential bugs. Or look at my Git history for the last 24 hours and just surface a quick summary for stand-up today. Amazing. Okay. So let's take a look at how the app did. It looks like it went ahead and... build something for us. It did some testing. It ran lint, test, build. Fantastic. And so we can go into the repo that I asked it to make. And let's take a look at what it's got. Cool. So I went ahead and one-shotted this app for us. There's a lot going on here, but I'll give a quick overview. The way that this is meant to work is we're going to put in a GitHub repo in maybe some custom instructions. It's going to analyze that GitHub repo, again, using hosted shell, and it's going to score it based on these seven metrics for agentic legibility. I wonder what would happen if you put the Symfony repo in here. Yeah, that's a great call out. So I actually had already loaded this up. Symfony is a repo that we just open sourced last week alongside this theme of harness engineering. I have previously tested this, so I actually know to tell the model. only score the Elixir folder. But Ryan, why don't you tell us a little bit about, oh, missing OpenAI model. Cool. So, I mean, I think we probably need to give it some environment variables. And let's say I should have these in, I should have these preloaded. And let's take another, oops. Power of live demos, folks. I know, it can never be. too confident in what's going to come back up. All right, let's try that one more time. I do like that the agent bias toward configurability, though. That's a nice touch. Very 12-factor app style. Okay, it's going ahead. Yeah, why don't you tell us a little bit about Symfony? So Symfony is kind of the next step that came out of the work that we did in this repo that is very bootstrapped for harness engineering. And it takes this idea that... If we have enough guardrails around Codex that it is reliably producing code that is going to be accepted by all the human engineers and agentic reviewers on my team, let's remove the humans from the loop entirely on poking at their terminal and instead have them working at a much higher level, defining work in linear. And Symfony is an orchestrator that manages advancing that work through the ticket queue, making sure that Codex is spun up in a work tree, implementing that code. putting it up for review going back and forth with ci and agentic code reviewers and only bringing the humans into the loop once all our previous constraints are satisfied and you know this allows me to truly focus my time on the hard stuff prioritizing work reviewing work and making sure that the work that we are doing is like accruing value to the actual products that we want to build yeah and it's really nice to be able to get be given a fully baked pr and be able to make a cheap yes or no decision on whether or not can merge. And then Symfony will manage merging it for me. Nice. And it looks like, yeah, this is reflective of the fact that you have put a lot of thought into those design patterns. Our app here gave you a B grade on your agentic legibility. Nailed it. There's a few things. So let's take a look at what it scored and its recommendations. So bootstrap self-sufficiency, basically Can the repo just get set up from scratch? Do you need to know? Is there external knowledge that you might need to know or commands that you might need to run to make this work? Task entry points. So is it easy for the agent to just run make or build or lint or that sort of thing? Validation harness. So how easily can it check the changes that it made? hard to know if you completed the job if you can't measure that the job is done exactly linting and formatting so i think this one um it found that you had a linter but maybe it wasn't as obvious to the model like where to find that and how to expose that um in practice i think linting is probably for me i think maybe the biggest like the best low-hanging fruit to tackle um it just saves i find in practice it saves like so many cycles of just like the model can just check check its work really cheaply that's right and it's super easy to add leverage to your code base by vibing up some new lids codex is really really good at this yeah um there's the yeah is there a map for the agent to take a look at where things are um are docs structured uh easily and then are there any decision records present in the in the app right um i think in this case this was the only one that that you guys missed on yeah artifact of open sourcing sorry folks But it ground all those, yeah, it made those recommendations. And you can see over here, it took, these are logs from the container, the hosted shell happening in the background. And if we go back to the Codex app, we can see like, yeah, we actually, in firing off this responses API request, we sent it a model. uh we specified this skill id which has all of that knowledge of the metrics in the background um and then we you know the team has actually put a lot of thought into safety and security around this stuff um it's it's not trivial to just you know sort of say like yeah let's open up all network access here uh so we did have to specify a white list of domains that we wanted the shell to be able to access yeah i think it's super nice how high level the container api is and it allows you to get very good secure defaults that empower the agent to cook while maintaining the safety and security and variance that you want because you know fundamentally uh these agents are going to do what they do and we need to make sure that we put them in the right environment to do that yeah um so uh yeah we'll have these slides here right these are the legibility metrics that we started on but ultimately um this was kind of a one-shot example right it was a single plan we one-shotted this we vibe coded this app think the more interesting question is how do you continue to build bigger things right how do you how do you extend that by coding session into into something much much larger that's right this is why I'm excited to talk today about harness engineering kind of what it is what we've learned while kind of operating on a project that my team took on for about five months where we had the hard constraint that no humans would write any lines of code so We ended up with a product that we're shipping internally today that is about a million lines of code where all of it, 100% of it, has been written by Codex. And this has been kind of a very interesting experience as an engineer here because I can't actually clack on the keyboard in order to make progress. I kind of have to step back and think at a systems level of how to enable my team of agents to go do the thing. Agents see the world and code slightly differently than the humans I would normally have on my team. So over the course of this project, we have been figuring out a bunch of patterns on how to refine the output of these coding agents so we can get to merge over and over again. And so one might imagine, one might assume that a million lines of code entirely written by agents that's got to be a hot mess right that's got to be you know like how do you manage that and you know make sure it's good yeah i believe the term uh for this is ai slop and one thing that's great because code is now so much cheaper to produce than before you can simply just say i do not tolerate ai slop which to me is slang for code i don't like right if you can articulate what it is about the code you don't like the next step is to write that down in documentation or in a bespoke code reviewer agent or in tests or lints if you can say what it is you don't like about the code that's being produced you can work toward encoding those non-functional requirements in your code base to just ban that code from happening in the first place one nice example i have here is one common pattern we see is that the coding agents like to optimize for local ease of use which means you can end up with several copies of the same routine in your code base one thing that has happened is we end up with very many copies of a bounded concurrency helper. But only one of them is one we've invested in to instrument with our open telemetry stack. So we have a lint that basically bans this function of various shapes from being defined anywhere else except our canonical async utils package. And this is just a vibe coded ESLint rule, which we can assert is 100% covered because We can just do that, which makes codecs exhaustively write tests for the positive and negative cases here. And this is kind of what it means to dive deeper into the systems thinking within your code base. Observe the mistakes or classes of misbehavior that the agents are making and do what it takes in order to just statically disallow them. This has been really, really cool because this allows us to actually encode what it means to have trusted engineering in the code. It's just not possible for AI swap to enter the code base. And one other neat thing is that these things have stacked incredibly well, right? We have kind of observed that as we have onboarded engineers to the team, they have a different set of experiences and ideas of what good, high-quality, correct code looked like, which means that each one of the engineers that has joined my team is able to reduce swap. in a unique way but because everyone is invested in putting that knowledge into the code base it means everyone else's coding agents have the best guts of everyone on the team um one neat thing is um oh yeah maybe we can go to yeah why don't we why don't we look at dig into some of that that code is usage yeah so um i think when we started this was very very slow right like Codex did not have a good idea of what my acceptance criteria was. So I often had to double click into a task to build some building blocks to get the refined output that I actually wanted. But once I had invested in those building blocks, I went from maybe a quarter or half an engineer's worth of throughput at the beginning to three to 10 engineers worth of throughput per engineer. And one other neat thing here is that as I have added members to my team, our throughput has actually increased because we have that flywheel of subsequent knowledge that every engineer has impacting everyone's coding agents for the better. Another neat thing here is because the only way to get code that is accepted is to get the coding agent to do it, we have had to pull a ton more context into the repository than I otherwise would have in the past. A good example here is I partnered with our security engineering team to upgrade our app to the blessed cryptography library that we support. But that decision happened in a Slack thread buried in our team channel two months ago. We hired a new engineer onto the team and they were mucking around in this part of the code and brought in a new NPM package for some of this functionality. I knew when I reviewed the code that this was not aligned with our engineering practices. But this is not the engineer's fault. It's not Codex's fault. That information just wasn't encoded in the constraints of our system. So I went back to that Slack thread. I did at Codex, please add guardrails to our code base, reflected that knowledge back in, and then redid the change, which then resulted in more aligned output because it used the blessed cryptography library that we were supposed to. Wow, okay. And so it sounds like then you're heavily relying on these integrations, these apps, right, into Slack or linear or whatever else to get context back and forth. Yeah, and it's super, super cheap to do this, right? At Codex, I go away for 5, 10, 15 minutes, and I come back with four proposed PRs on how we can make our code higher and higher quality in the future. The other pattern that I think was really, really neat is We have a lot of context in the code base. Too much to put in an agents.md file. Just talking about our security best practices is a 250-line markdown file. Same for what it means to write reliable code. I'm sure all of you in your past have gotten a page that was resolved by adding a retry and a timeout to some network code. And we all know this, but still, code ends up in production that does not have retries and timeouts. Code produced by codex is the same. So kind of writing that non-functional requirement down, giving the agent pointers in agents.md on how it can look at our best practices for reliability, and then reflecting that back into the agent in a reliable way via a code reviewer operationalized around reliability who leaves comments on PRs that our primary authoring agent is forced to read means that we can kind of steer the system. to converge around our set of guardrails in a way that's actually higher reliability than having the humans do it codex is very very patient and willing to take as many code reviews as we can throw at it whereas i might get frustrated and just yolo merge it anyway so we do end up with very high quality code as a result of this here another neat thing is this stacks very very well we had a new engineer join our teams very product-minded and was frustrated that Codex was not able to effectively smoke test changes to critical user journey flows in our app. So pointed Codex at the code base, asked it to spider through all of our user-facing features, write product specs for them, write manual QA plans, write the user needs that these flows address, and then pushed into a bespoke reviewer agent to look at every PR and generate a manual QA plan, which again, Codex is able to look at. in order to validate that it's changes work end to end and are aligned with our apps functionality and what i find really fascinating about this sort of setup is you know normally these things they maybe exist in slack or in conversations but we've never sort of had the ability like we've never taken the time to actually write them all down and encode them in our code repositories but to your point about it is now so cheap to to generate these things right that that is kind of a new avenue of documentation yeah if you want codex to be an effective member of your team you need it to know all the things it takes in order to have acceptable code in order to merge it really is about the non-functional requirements in order to get truly aligned output from these agents So I talked about this a little bit here, but as we have onboarded new folks to our team, they each bring their own experience and expertise. I myself bias more toward back-end, infra, and architecture, which means I just did not have the know-how in order to get high-quality React code out of Codex. But as soon as we hired someone with deep front-end architecture experience and they started reflecting that knowledge into the code base, now all of a sudden, hooks were getting decomposed to a single hook per file we were getting small components that were easily testable via snapshot testing and composed together well to have smaller files that are more efficient for the agent to page into context and with each subsequent engineer on the team they encode more and more of their taste and expertise which means everybody's coding agents get more effective that's so cool so Some key tips on how you might apply some of these patterns in your own work. You should check out the Codex repo, which already has much of this in it. Just clone it and ask Codex to explain how it is structured and to teach you about the best practices that the team has used. Another cool thing that I'm seeing all over X right now is folks are literally taking the link to the Harness Engineering blog, giving it to Codex in their repository, and say, make my code base more agent legible. Make it do this. and folks are getting good results that is improving the autonomy quality and long horizon task execution of codex uh super super cool that we can essentially ship prompts to the world uh to help uh everyone be more effective with codex yeah and i love this it's a shift like we're saying from from you know coding in on the keyboard to now orchestrating right at the next level which symphony that's right that's how we get to the repo name that's right All right. Then I will hand it over back to Christine. Awesome. Thank you guys for walking us through that. Now for our customer spotlight. Really excited to welcome Mitch, the co-founder of Basis. We're going to pop out of this screen share and then Mitch should be joining us on stage. There we go. Hey guys. Thanks for having me. Hey, Mitch. I'm thinking I'm going to be saying a lot of similar things. Perfect. You were just muted there for a sec. OK, great. I was just saying, I think I'm going to be saying a lot of the same stuff Ryan and Co have been saying on putting stuff in the repo and whatnot. But this will be a fun discussion. Let me see if I can share my screen. So I had Codex make a quick presentation here. You guys see that? Yeah. So I'm just going to quickly go through. talk about some some of the stuff we do which i think is very similar to some of the harness engineering work um and uh both on how we do it for engineering in our code base but also actually at the company level um and how i think you can think about using codex beyond um just even the code base so i think the the first thing that's really important um and let's talk about the code base first and we'll talk about that company after is actually and i realized let me give a quick introduction of myself so i'm mitch i'm one of the two co-founders of basis i um basis essentially is an agent platform for accountants we recently raised a series b and we are very very focused on um engineering our company in such a way that we can move extremely fast and the reason i always tell people this is so important is that our ambitions are so large that if you actually think about the scale we need to be at in very short periods of time it would literally be beyond the laws of physics to be able to hire that many people And so we can't rely on that. Instead, you have to really engineer your company and your code base to be able to have as much output as if you were a company that was 10x the size. So going kind of back onto the code base side, doing that, I think, really does require a big mindset shift. And part of what you guys just heard about from the Symfony side is people need to be able to shift from doing to managing. And I think that's actually a very hard, um paradigm shift to make and different people uh are able to get to it at different kind of uh different speeds but that shift doesn't really it's hard to make people make that shift whether it's engineers doing it or people on the sales team or or the post sales team if uh the agents aren't actually working well so it's hard to go and convince someone hey like you know go you should just have let's say this agent write the code if the agent isn't actually writing the code well number one and two you have to have the mindset shift about how do you build for agents not just for people right you want agents to have the ability to self-verify you want them to be able to um uh understand like what the different intents are uh that people are are giving them and even then if you're in a production code base or in a larger company you need abilities to collaborate so that one person's preference of how some agent behaves doesn't now go and override another person's preference and so you really have to go and build this up so what do you actually need to be true at the code base level for this to work? You need to have good standards in terms of how you want the code to be written. That needs to be turned into good context. And then you have good processes around this to make sure that context is kept up to date. And then you need internal setup that's easy to operate, right? If you have a bunch of dev tooling, then none of which has MCPs or ways to integrate in with your agents, then they're not going to be able to actually validate the work or use the same tooling that a developer would. So we, for example, at BASIS, we have something called Satellite that we made, which is a single MCP that essentially wraps all other MCPs so that developers only have to integrate with one MCP rather than having to integrate with 50, just as an example of that kind of thing. Why is Codex really good at this? So I think specifically for production code bases, if you want the agents to be really good, you need to get really good at instruction following and codex is very good at instruction following to the point that actually it forces you i think to be even more precise in your uh documentation and more precise in your context so that um codex follows right instructions you need to be codex is really good at gathering context ahead of time right right i was talking earlier about like the progressive disclosure that's you know if you're going to have lots of progressive disclosure which is really important for any level of complexity you want the model and the harness to be good at um uh going through that progressive disclosure to gather the context it needs to do tasks codex is excellent at that right and so it'll constantly go out and and do that and um you need to be good at just making decisions uh in that areas of ambiguity and codex is really good at that so let's talk a little bit about basis and i'll go into a couple demos here of how we do this in terms of how do we kind of set those standards so we focus on a couple things we focus on agents mds uh so we have an agents md at the root And then we have agents.md in different modules throughout the code base with the kind of principle that if the information is specific to that module, then you can put it into that modules, agents.md, versus if it's maybe generic across modules, you would put it into a skill. And let's see what else I have here. Yeah, so I think it's super, super important that you set these standards, you treat them as... kind of canonical, and then you build the processes to keep them updated. So maybe I'll show you one quick example here. I'm not going to show, I'm going to have to be careful here, given it's a live stream on what I can demo, but just to show, let's say here. So this is a skill that I have. You can see that agent skill. It's called paper updating. I'll explain what paper is in a second here. But this is a skill that we have on how to go and update something internally we call paper. And the key here is that there is an owner right there who is put on the skill as part of the front matter. And what that means is that you can have very distinct responsibility for who is responsible for different skills. So that, for example, let's say you were running an agent that was going to go and look at the different skills and see, hey, are there any skills that have descriptions that conflict with each other? uh making sure that the um the skill that if it knows who the owners are so you can like slack them or something like that and then which gets to this internal tool that we've built called paper which i'll just show you for a second here um so paper is essentially a way to if you're going to have all of the context in the repository which we want to do and our our monorepo is called arnold because that's that's what we called it um fun fact when i came up with that it was because uh arnold was the most popular name of an accountant in the us according to chat gpt in like 2023 so that's why it's called arnold um so if i go to paper and just show this to you quickly um so here is is paper and like as you can see we have you know code based engineering engineering at bases i'm not going to show you all the different skills but i already have it pre-set up the skill i was just looking at here so it's very clear who the owner is what the flow is and you kind of go and sort of see it and this allows you to really easily at least for us um be able to uh make sure that you're not getting slop into the code base that you actually are keeping it updated that people are actually looking at the skills and uh able to kind of go and read it so that's just like an example of that of of kind of do how we have skills and then we also use sub agents codex as of recently supports sub agents codex is really really good at enforcing standards for sub agents we use sub agents very heavily actually um so some of the sub agents that we use is we use um uh We have a specific subagent whose job it is to enforce standards. So Codex, it'll do a pass, and then it'll go and call the standards enforcer subagent if there's any thing that Codex might have missed in terms of our standards. We have a subagent for helping babysit pull requests in case Codex wants to send someone off to go and do that. And so subagents, I think, are a really, really good way to take flows that Codex is amazing at, but you want to be really sure and put that in a specialized sub-agent that Codex can go and call. See what else we have. So a couple other quick things here, then I'll go into a couple more quick demos. So I think it's really important that agents have the opportunity to actually verify their own work. So at the end of the day, you need to be able to have if if you want you know codex is really good for going for long periods of time and one of the things it's really good at is using tools in context to sort of make sure it's doing a good job so you need to give it ways to do a good job right and that includes tests can it quickly run tests against the work it's doing so that it can validate itself one flow that i like doing a lot is having a more test driven development approach to things where you don't just have a product spec but you actually have a list of tests that Codex can go and generate for you that outline kind of the intent of the change so that Codex can go and ensure that the different user flows that it is making actually work by the time it comes back to you. Okay, and lastly here, I think one of the big things that starts to happen once you actually have Codex, let's say, writing all the code and is able to go and execute things really well is that decision-making starts to become the main bottleneck on your velocity. So we have been trying to do things on our setup that allows for decision-making to actually start to become faster as well in working with Codex. So one of the things we do is we have this concept of .notes. And .notes is essentially a way where if you're working with Codex and you have some decision, and let's say you told Codex, hey, actually, you should be doing it like this, Codex will put decisions that were happening throughout the session. It will put those in .notes. which is like a part of the repository just to streamline decisions. You can sort of think of it like commit messages, but commit messages that Codex can write to whenever it wants, rather than only when it is making a commit, which creates a full kind of history for every commit of what were the decisions made in that session that you can use as a historical kind of debugger. So if you're going back to a historical commit, you can say, well, wait a second, why did... Codex or whoever made this commit implement it this way, not this way, you can look at the note in the .notes folder, and it will tell you that. And obviously, specs that live in the repo is really, really key. So as a quick example of that, I'll show you here. I have a product spec that I put for paper as kind of an example here. If you kind of go to GitHub, it's sort of a little not exactly the greatest, easiest thing to read. It's hard to leave comments. If I go here, it's, you know. it looks okay and so one of the things we built internally because we have this internal app called paper is the ability to go and collaborate on specs right inside of your code base so i can go um here for example and this is now the the spec i'm looking at i can go and leave a comment test codex comment and this will now go and leave a comment that is actually directly in the um in the inside the code base and so all of the um the commenting all of the resolutions all of the interactions all of the planning actually happens literally inside of the code base you see now it says refresh um literally have inside the code base and if you go to here you'll see the comments so it's just a cool like internal tool we built but the reason for this is specifically for the purpose of uh as kind of ryan was talking about bringing stuff inside of the code base so that we can have as much context there as possible so that Codex has all the context it needs to go and help us make decisions and then implement this. And then lastly, and then Christine, I'll hand it off to you, is I think it is really important to not stop at the code base. So you'll note here that I didn't have time to go actually give Codex the screenshots I told that it would give it. For the company, we think about this kind of mindset of having context in the code base, not just as a production code question. We think of this as a company context question. So we have two repos in our company. One is called Arnold, which is our monorepo, and the other is called Atlas. And Atlas is the monorepo of all the company context that is not in our, the production repo. And for example, things like, if I go here, things like, where was it? Things like, so here I'm in the Atlas, not Arnold. This is in paper again. But like here are some of our operating principles. um which you can go and you know look on our careers page for example um and our operating principles you see like who the owner of that is and having that very very clearly in a code base because now say you're trying to work with codex to help plan well codex can go and help and read all the information about the company in order to help um make decisions about different things and so i think once you start realizing this you can do a lot more and so may i'll just show you one quick example what that allows you to do let me quickly share it different part of my screen here if i were to go and actually pull up codex um let me pull it up and i am uh since we used the app earlier i'll use the cli quickly um so in this case this is the me making the powerpoint earlier um so if i'm here like i for example have a skill in my personal that's called start my day and what start my day will do it's going to run through my morning routine It's going to grab all the context from the last 24 hours, refresh a bunch of different things, do weekends, et cetera, et cetera. So I'm not going to actually let it run because then it'll show some information, but allows you to kind of go. And it can do this really, really well because it has all of the context that it needs, both from my personal notes, our kind of company repository, and from Arnold. So I'll pause there and give it back to you, Christine. Awesome. Thanks so much. So we have about 10 minutes left for Q&A. Mitch, if you want to stay on, we had some coming through the chat specifically for you. And we'll just back into the screen share. Cool. Okay. First question, Mitch, how did you use Codex to generate the deck? Yeah, you can go and have it make PowerPoint in Python, but that was just HTML. So I just told it to make HTML. And I... voice, voice moded, all the context about the thing I wanted. And if you notice it used, I think it used opening eyes fonts. I told it to go find what the opening eye fonts are and find a similar one. I do this all the time too. It's super good at just vibing a bunch of single page react slides into place and doing a pretty good job at it. Oh, totally. Yeah. Nice. Okay. Next question says, shout out to Mitch. Met you at an event in New York City. This was awesome. A few questions. So is there any checkpointing or the ability to revert? And from a management perspective, is there any plans for collaboration features like shared threads? Can Codex execute browser previews or browser automations? I can take that last one. So Codex can definitely execute browser automations. There's a Playwright skill that's been around for a little bit. And then we just released a Playwright interactive skill, which I think takes it even further in terms of the depth that Codex can go in using the browser. Personally, whenever I'm doing front end, if I have feedback for Codex, if it's gotten a box wrong or an alignment wrong, I'll tell it use Playwright and just keep iterating and screenshot it until you feel confident that you've got it correct. I think the other one's in terms of checkpointing and shared threads. I'm not sure about shared threads. I think that's a great question. In terms of ability to revert, in the app, you have the, like everything shows up as sort of staged files. So it's very easy, or unstaged files. So it's very easy to just click and sort of stage stuff. Or you can simply tell, you know, codecs or have it in your AgentsMD, like commit everything as you go, right? And I think once you set up that scaffold, it can take care of itself. Yeah, I definitely like Codex Commit its own work, and I give it a skills for what our expectations are around commit messages and the progress and the granularity of those commits. Works pretty well. Okay, next question. As we move from AI pair programming to full agent delegation, what are the key design patterns that make an agent reliable enough for production systems? So what I've found here is it is... highly beneficial to put in pretty rigid architecture patterns that you would expect to deploy in a company of a thousand or ten thousand engineers which means uh i have dived really heavily into separation of concerns proper package layering making sure that business domains have high cohesion and encapsulation and boundary separation and this helps with context management right if you're working in one set of business logic and the agent can take as given an opaque interface that it can rely on invariance for right you know for example if i am building a um you know a task summarization feature as a core set of business logic if it can accept a dependency of inference right it doesn't have to deeply understand it i've limited the amount of context the agent needs to page in And this is sort of the same pattern as progressive disclosure that we're doing in the docs. So being able to structure the code base in a highly opinionated way also permits those code as prompts to follow the same sort of pattern. We, by the way, did far more refactors, I think, than you would have five years ago because of this. So I totally agree. Okay, next question says, how does your team maintain agent instructions? Does each engineer provide their own personal set of instructions or does instructions get shared slash version controlled? How does the team sync up on adding or improving instructions? So everything lives in the code base, which means at least on my team, we have very few personal agents.mds or personal skills. and we treat this as a way to encode leverage for all the agents that are operating on my team's behalf also we have not we have collected around only a handful of skills that are pretty general right we have a skill for making prs we have a skill for landing changes and deploying them we have a skill for making commits a skill for doing code review a skill for analyzing the architecture of proposed plans and really that's it Any additional leverage we want to give the agent either goes into reference documentation that goes with those skills, scripts that go with those skills, or the documentation and tests in the repository itself. And this has the benefit of getting things out of individual teammates' heads and into a space where everybody's agents can benefit. In terms of sharing knowledge, we have bias toward really high-frequency in-person stand-ups. we have 30 minutes stand up every day which is maybe something that's a little bit different from how folks have been working recently but the code velocity is so high and each of the engineers is disconnected enough from the code that it can be weeks before i realize like core architectural patterns have changed so bringing those to a synchronous time for the humans to collaborate has made a ton of sense for us uh miss is there anything different that that the basis team does here No, I think it's all the same. The thing I'd actually be curious to ask about is the thinking between putting some of those front-end or back-end standards into docs versus a skill, or it's maybe just kind of marginal. But we put a lot of that in skills, and then we have docs as meant to be, docs kind of meant for humans, and then obviously still useful for agents, but we, for now, keep a lot of that in the... in the scale so there would be like a front-end scale that the agent you know codex anytime it's touching any front-end code will go and look at it and that will have a bunch of references under it that will uh you know have specifics on different parts of the front end depending on what it's doing um assuming that it's not completely scoped to um like a module in which case it'll be an agent 70. yeah we have uh instead put all of that uh task specific expertise in the documentation hierarchy and instead focus the skills on high-level operating modes. So we'll have a principle review skill that the human can then steer to look at specific documentation, but is otherwise just a general mode with which Codex should be doing the task. That's cool. Worktree sounds cool. Can you demo that quickly? Sure. Yeah, let's take a quick look. Let's go back to Codex. So the way the work trees work, so right now we just pick a new work tree. And then I'll just say, you know, modify the plan MD to be shorter. And the idea here, I'm going to toggle off plan mode, and I'll put it on low just so it's fast. But you saw briefly there, created a new work tree. And then now it's going to go ahead and... start making uh yeah it's smart enough to know that it should read it um but it's going to take a look uh and i think it's looking for which one it should edit right because we previously had some archived ones but basically one thing that didn't happen in this branch is we didn't bring in automatically our existing diffs right so like all of the changes that happen here locally are currently separate from this work tree and so that way i can work on this branch there's an option to bring those changes if you want them but by default you know it'll it'll take the clean main checkout or whatever branch you want and go ahead and do it so it's going to modify these these files and then we'll have the ability to either spin off a branch and create it later or once it starts making diffs and if you're on a separate branch it'll decide whether you want to hand those off to to your local environment Yeah, the way to think about this is I can have multiple working directories with multiple different branches of the repository on my file system at the same time without having to clone the repo multiple times. And these commands are pretty fiddly. I hate using them. So it's great that Codex app is just going to do this for me. So it made a bunch of changes here. But those are not going to be reflected in our main thread. Okay, next question. We'll just rapid fire through these as we're almost at time. Due to progressive disclosure, should you just enable all skills so they can be called if needed? I would say yes. The thing to invest in is short, high-quality descriptions that live in the front matter of those skills, because that's going to kind of be the teaser that the model gets to determine whether or not it should invoke a particular skill. So we kind of get to go from... a short little snippet describing what capabilities are available to the detailed instructions the model should follow yeah we actually have a codec sub-agent whose entire what it does is it can uh spin off um like it could spin off version of the sub-agent that will test for different skills so it'll like give examples of different instructions uh and see if it actually triggered the skill if the sub-agent grabbed the skill or not as a way to kind of like pseudo test pseudo test if the front matter descriptions work well And then Ryan, what about specific skills in our library? Which library? This was just a follow-up question for the App Store. Oh, I really like using the Figma skill. It's super, super neat to be able to close the loop from work that other members of my team are doing to pull that context into Codex. In the same way, we're doing all these tricks with Slack to bring context into the code base as well. Nice. Okay. I'm curious if any work was put into creating a bootstrap for harness engineering on a Brownfield application, for example, documentation management, generic rules and conventions, and if not, what sort of specific implementation details could you advise us on? Yeah. So I would say we are still actively figuring out what it means to deploy these techniques into a Brownfield code base. And this is something we're super happy to collaborate with the community on to figure out what works. In general, the techniques we've been doing are to carve apart parts of the repository into isolated business logic domains, add interfaces, add high-quality local set of documentation on the best practices that every member of the team that owns this code is looking for. So we have been having individual docs subtrees that are close to the code that's being modified using that same sort of structure that we lay out. And yes, turn on all the lints you can. Okay. Do you have any resources or examples for these MD prompt files to help guide codecs with long-running tasks? I mean, I'll let my answer, but I think all the skills that we've had, the metrics that we've shown, and the code for the sample app, we're going to have that in the build hours GitHub repository. So you can take a look at how we're asking the model to grade these MD prompt files. Yeah, and I'll drop those links after this session and in a follow-up email. Mitch, there were some questions for you, too. People were really interested in your slide deck as well as paper. One thing I'll say is to see these files, what reliability looks like for your code base, what security looks like, what front-end component architecture looks like, I'd say just sit down as a team and rapid-fire throw some bullets on a whiteboard. around what you think good looks like and these can be pretty high level right like something like secure code has secure interfaces that are impossible to misuse is just something you should write down uh and just getting a couple of files in the code base that have the seeds of this makes it really really easy to incrementally add to them so take the time as a team to figure out what good looks like for you cool um yeah like i said it'll be in github um Awesome. And with that, we will wrap up. Here are some more additional resources, but again, I will send them out post-session. And then just a quick plug on upcoming build hours on the next slide. You will see we have two coming up, one on agent capabilities that's happening March 24th, and then GPT Real-Time 1.5 on April 15th. Follow this link below. That's our homepage. You can find past recorded sessions as well as sign up for everything that's coming up. And with that, we will wrap up. um thank you so much for attending and thanks mitch for joining us have a good one thanks fam happy building

Build agent-powered workflows for real engineering work with Codex and OpenAI APIs. In this Build Hour, you'll learn how teams are moving beyond pair programming to agentic delegation, where AI systems can take on entire engineering tasks from planning to execution. 

Charlie Guo (DevEx), Ryan Lopopolo (Future of Work), Mitch Troyanovsky (Cofounder, Basis) cover how to:
• Ship new features faster using Codex, with real examples from Basis: https://www.getbasis.ai/
• Build agent-driven systems using new API capabilities like hosted shell, skills, and websockets
• Evaluate and improve agents using the Agent Legibility Score framework
• Apply Harness Engineering techniques to make agent workflows reliable in production
• Use GPT-5.4 for large-context reasoning, computer-use tasks, and tool-based agent workflows
• Reduce manual overhead by turning complex logic into reusable agent-driven flows

👉 Codex docs: https://developers.openai.com/codex
👉 API docs: https://developers.openai.com/api/docs
👉 Harness Engineering Blog: https://openai.com/index/harness-engineering/ 
👉 GPT-5.4 Blog: https://openai.com/index/introducing-gpt-5-4/ 
👉 Follow along with the code repo: http://github.com/openai/build-hours
👉 Sign up for upcoming live Build Hours: https://webinar.openai.com/buildhours

00:00 Introduction
01:58 What's New: Codex
05:01 What's New: API
08:33 Demo: Agent Legibility Score
21:53 Harness Engineering
32:32 Customer Spotlight: Basis
47:57 Q&A