"Coding was never the bottleneck" | Darwin Wu, Inngest

The other thing is that if you don't have a test harness, if you don't have something to guarantee quality, like you rarely see any of those actually go out into production, right? You can just spend a bunch of tokens and doing a bunch of things, sure. But is it actually going to be served or are you just wasting, you know, electricity? Welcome everyone. I'm Mateo from arcade.dev and I have pleasure to be interviewing Darwin. He's a tech lead at Ingest. Ingest is an engine for durable execution for your agents and every app that you can imagine. Welcome, Darwin. Yeah, thank you. Thanks for having me. All right. The way this interview is going to be is I'm going to go a little bit in your specific history, then a little bit about Ingest, and then your opinions about agentic stuff, MCP, and things like that. To start with, you came into Software from an initial angle, a political science and economics major who fell in love with code. And you've also got a habit of building your own tools. from the community Elixir SDK for ingest to your own Emacs slash Nix. What pulled you toward durable execution and orchestration as the problem worth working on? Yeah, this probably goes back to, you know, experience. I've been in the industry for roughly like a decade plus or just one more at this point, and I'm not really counting at this point. But every company in the past that I work with always has some kind of workflows, right? They have to build it. Ruby has a pretty famous one for sidekicks. Python, a long time ago, I think it's still used. It's like Celery. These are all open source solutions. And you have things like Revit MQ, who works as a Q, and then you have Kafka. So there's all these things that have been designed to, you know. use cues or some kind of durable guarantees for things that you want to be able to publish and consume at a different rate. And every company have their own solutions because most of the time you just end up patching a bunch of different things and glue them together to make it work. So you have to provide a queuing mechanism, but how do you observe it, right? How do you make sure things, everything is not going to be as simple as just you dump something in and consume it on the other side. You have certain things like, how can I stop this? Or how do I chain this? And then now you're going into thinking of like different queues. Like, you know, queue one, do this, queue two, and then change to the next queue. And then you have a different worker on there. And then you start spending more time on the infrastructure layer. on what to do than the actual problem you're trying to solve. And that's kind of like, you know, it's to an extent, I guess, put it in another way is that this problem has always existed. But the framing of the robot execution is more recent, but more in a holistic way, because previously, it's just more scattered. And I've kind of been working in this space for a long time. It's just more that now we actually have something like a name to point to. than just talking about, oh, I work on backend infrastructure, platform infrastructure, and just that you don't really know what you're talking about. Amazing. For the audience that don't know what durable execution means, can you give me a two-sentence summary of what makes execution durable versus not durable? Yeah, so durable really just means it's persistent, right? And what needs to be persistent? is certain things that you probably don't want to lose data on it. Or certain things like you have someone joining a meeting and you get a notification of this. You don't really need to persist that. It's just like, you know, real time, if you drop that notification, it's not that big of a deal. But if you have something that you need to schedule a payment and just schedule a billing, like invoices, you don't probably want to drop those. Sending email might not be that critical, but there's definitely a different level of severity in terms of what should be lost versus what shouldn't be lost. Durable in this sense means that for the things that you really don't want to be lost and you want to make sure that goes through. So yeah, at least that's my interpretation of it. Amazing. I think like, you know, going into Ingest now directly, Ingest's positioning in the website is unbreakable agents, invisible infra. And it has this philosophy that durability belongs in the code and not really in the infrastructure that you bolt alongside the code. Can you unpack what that means and why you think that's the right way to think about the normal execution? And, you know, especially, you know, in the agentic era where everything and how we do things keeps changing every week. Yeah. Let's see how to, how should I put this? And the part that Ingest is built around is we want to extract the heart problems away. So things that you probably don't really need to care about or you have to, or like something that you want to spend too much time on and just focus on the problems that you want to. And we kind of have this in different technologies, right? Certain things like transactions in databases where you want to make sure things are ACET and they don't conflict with each other. Those are built into the databases and relational databases and such. we're kind of a little bit taking approach from that side of things, at least when I started three years ago when working with Tony and we're wondering what we discussed is the idea that the durability of the step, like a user probably don't really need to care about how it's persisted and how that data is cached, is it executed just once, is it executed multiple times? And you would rather, if you want to, when you're building a workflow, you kind of want, as a software engineer, I have a rough logic in mind that how I want to build this and how do I want to compose the steps to make this work. Right. But, and we're trying to tailor towards that side so that people can actually focus on the logic and translate that into code, not necessarily translate into code, translate to the code based on the infrastructure you spend on. So that's where our focus is. On the developer experience side, I mean, one way to phrase it is just we focus on the developer experience, but the more intention behind it is that so that people can actually focus on what matters to them. Then, you know, going back to, you know, just like you don't have to care about is it RevitMQ or Kafka, right? It's just like, hey, I want this thing just executed exactly once. This is step.run. Like if you think about it, transactions, this is kind of like transactions databases. these will be executed once, it will be persisted for the next, the day will be persisted for the next one to consume, and you chain it from there. From there, is it like, you know, just a function that with multiple steps, or does it need to fan out to multiple functions to kind of do, you know, a map reduce style of operations? It makes it a little bit easier if you want to think about it, because it's fan out, it's fan in, right, if you want to pause something. The example that people give, actually, that's something I give when I'm onboarding people or when talking with new hires, too, is that you think about email flow. You will sign up for a company. And from a company perspective, you want to probably send up a welcome email. And then maybe there's some kind of activation that you want them to do. So you kind of put in the email. But you kind of want to wait for three days or so before. to see if they actually did it. If not, maybe send a reminder. But if they did, just maybe send a follow up right away. These are the conditions that you have in your head that you can probably reflect it pretty well in code. But if you think about how do I do this queue, wait for three days, what does that mean? In queue something they need to pick up in three days. And if it's done before then, do I discard that thing? these are the things that probably people don't really want to think about. And that was where it came from. That's amazing. And I think, like, looking through Ingest documentation, I see that you have abstractions like step.run that deliberatively collapse queues, PubSub, idempotency, and state management into a single call. How hard is it to do all of that into one single call that will run at, you know, more than 100k execution per second without leaking complexity to the developer? Yeah, I'll say we're getting close to millions at this point, executions. Yeah, some of our larger customers are actually in the billions in a month. So not seconds, but still per day is like hundreds of millions, easy. But yeah, this part is actually a lot of distributed systems in terms of engineering. It's understanding constraints, right? And then, because we're building abstractions, so we want to make sure the expression works well. And the hard part really, it has changed differently based on when we were three years ago, when we are now. Three years ago, it was really more just, is this the right abstraction? We don't think about is this the most optimal of doing it, it's more just the explorations. And a lot of this is around the interface design, because we want to make sure that we get a good interface. Like good is obviously subjective, especially when it comes to developers, but a good default that is that when you look at it, you understand what it does. without having to think about too much. And it's just something like the nod that you can turn on. So there's a lot of this is focused on initially on that. But now it's actually just making sure everything works well. Because I mentioned executions are in the millions, but our queries per second for some of our subsystems are all easily in the millions. And it spikes as well. So now it's actually really focused on making sure that things scales. well, operates well in scale, and then also just continue to expand that to make sure we can handle the growth. And yeah, hopefully that answers the question. Absolutely. And I wonder, since, you know, three years ago to now, I mean, at least the way I see it, a lot of what changed is people are starting to use agents like crazy. And that's... in my opinion bound to increase the number of requests dramatically, right? So how do you see these like durable executions relate to agents or how do agent designer needs to think about durable execution? Yeah, this is actually an interesting topic. And I think like overall, the industry I think has started to get, you know, a little bit more realization into agents are fundamentally unreliable. Or LOMs are generally unreliable because your call can get cut off midway, right? And then it's like it does something very undeterministic. This is undeterministic behavior plus distributed system problems all batch together. So, to that extent, if you look at early days on how, you know, like LaneChain was essentially like an interface for, like, LLM providers. But they kind of, like, if you look at how they changed their pivots, it's like they're kind of moving towards, you know, durability, right? Because at first, people mostly just figured that it's like, oh, we're gonna make all these API calls, and it's gonna work great. And then, like, they learn in the hard way that it's not gonna work great, because now you have to retry things. And you potentially probably want to constrain your API calls so you don't get raided in it. And all these things that you, like all these great things you have to work with, but they're pretty provider and essentially those are just like workflows, if you think about it. Workflows in a way that you want to flow control them. So that's where, to that extent, is it a surprise? No, not really. Especially if you're in this phase and building something as an agnostic system, then it kind of like if you look at it, it's like, oh yeah, this fits great. I think the main question is really just that, how do you make it integrate well? And among all of these unpredictability, people want some sanity, right? business don't operate on unpredictable outcomes. You want things to be, some things can be unpredictable as long as it provides value because a lot of things are value-based. But you still want some sanity on making sure that this is within the range that is acceptable, not all of a sudden just spit out random stuff that will damage the brand. So to that extent, that's how we start integrating. with agents. And I think the early question in terms of the increased load and such, that's the part I kind of mentioned earlier that we're working on, and making sure that this still continues to scale well, because that part is actually just purely a distributed systems problem than anything else. And it's really about, you know, understanding where our bottlenecks are. How do we remove those bottlenecks? How do we engineer ourselves out of those problems? And I guess in that sense, it's just like as quickly as possible before an issue shows up. Yeah. Amazing. So that's fascinating. Yes, I agree that agents will behave non-terministically most of the time. And yes, having some sanity is exactly what we need. And a piece of that, in my opinion, is also this idea of having human-in-the-loop interruptions, right? If the agent is trying to attempt something that is going to be useful, probably, but it requires permissions from the human, you need to stop it, present the request to a human, and then continue as if the session didn't actually interrupt. Why do you think that patterns matter so much? And what's tricky about getting that to work at scale and reliably? Yeah, so this actually probably goes back a little back to the sanity conversation. This is a way for it. I don't know if I could approach it in a way, but it's probably just like human nature, right? You want to be able to control certain things. Certain things you want to delegate. It depends on low leverage. The leverage is the low, there's the high, et cetera, et cetera. But ultimately, people want to make sure they can control. And then to that extent that what context this means is, especially if we're talking about apps, there's different types of damage that you can cause by unpredictability, right? It's like if something that is supposed to be very professional, something starts spitting out, you know, things that are very non-professional, then that is gonna be a brand damage that, you know, just like... come out of nowhere. And you want to be able to control that. And this is, that level of different level of control is definitely different based on the industry you're in, you know, based on the category you're on. And that's part is, then that's what, you know, at least from my observations is where it's coming from. It's just that that's where you can at least inject your opinions, your thoughts, or quality control, maybe if you put it another way. to make sure that you constrain that unpredictableness. I don't know if that's even what to put into. And working at scale is actually interesting, right? Because you, it goes back to a little bit on the dual work execution too, is you kind of want to pause something until a human reacts to it. Funny enough, one of our APIs is actually built for that, really. It was actually initially built for typical app cases where you can think of cases where the workflow proceeds to a certain point and then you're waiting for human approval. And if you look at it from a CI, CD perspective, maybe a long time ago, certain things that Spinnaker has that, whereas your deployment pipeline goes all the way to step five. And then from there, you have a human approval to get a fully out release. are certain things like Cloudflare has, has different cases internal deployments, where most of it is automated, but certain cases you need a human approval to let it through. So it was initially designed for those type of cases, but turns out, it works pretty well because agents are workflows as well. So now you're basically just like this one more case that just fits into what we have. And the API I mentioned is talking about the wait for events. Essentially, it's like you pause the execution and you halt the execution until an event comes in. And that event, and because we, I guess I didn't really mention this, but Ingest is also essentially an event-driven system. And we model Ingest as events because that's what happens in the real world, right? Everything is an event. You're clicking a button as an event, you're sending out an email as an event. someone signing up as an event. So to that extent, wait for events is really just like, okay, let me pause this until something comes in. And that something in this case is a human approval. Or if you decline, it just stops. So it's like, so yeah, that's how it is, I guess. Getting that to work at scale is interesting. I don't know how much secret sauce I'm not spilling, so I'm going to try to balance that a little bit. But that part is actually one of the most interesting engineering problems we have. We have very high loads, high throughput problems. This part of the system is actually very like algorithmic complexity in that sense. It's like we're actually building out ASTs in memory, like Fset String Textures to make sure things are parsed really fast. Or, you know, how do you make sure things are discarded really fast so you actually don't actually spend CPU cycles on evaluating things that it's like. not needed. So that's actually on the scale side, where is things out of the box doesn't work, so we have to have like custom solutions on it. There's actually a blog post on NGIS if you want to look at it. I think it's called Accidental Quajillion. Yeah, this is actually about the expressions, how we evaluate the expressions to make sure wait for event works well. And fundamentally, this also applies to what we mentioned about the agent in the loop. Now, if you have hundreds of thousands, we have to process millions of things in milliseconds. And that's the interesting part. There's a couple more improvements we can make that we can go into, but I think there's cases that we have to go into actually doing bit shifting to just gain one more milliseconds. Makes sense. I'm very curious about this. Do you find agents I mean, you describe it very well as agents are workflows. And do you see the failure modes of agents similar to previous, you know, previous generation apps or they fail more or less or are the types of failure different or meaningfully different? Interesting point. I would say like to an extent, I would say 80% is actually probably similar. Similar in a way that is a human error of encoding your intention error kind of meaning. Because code is like if you implement it wrong, you have a bug, it fails, right? If you're in a workflow, it probably retries and then you deploy a fix next time it goes through. Agent to an extent is kind of similar, right? You accept that you're iterating a lot more and then you just like you don't exactly know how it works sometimes. So your system problem might change. your specific prompts, like you tweak your system prompt to make it a little bit more efficient, you're trying to experiment, like doing, trying to do something like A-B testing against different prompts or potentially even like different agents or different LOMs. And the failure model in terms of that is, you know, it's an external API. So your rate of failure with an external API is not that significantly different. So your error rate will go up if open API is having. Sorry, OpenAI is having an outage, or Cloud is having an outage. That part is probably going to be the same. But like from the intentional encoding side, I think that's not going to be different because at the end of the day, it's still the same humans writing that thing, right? And I don't know, like from my perspective, at least humans are not the best things, the best beings at getting things right in the first attempt. So what you said, actually, I want to push a little bit further on that because I see this pattern now that with things like ultra code and like code mode and things like that, a lot of the execution will be done by a sub-agent that writes some code, right? What's like, if those pieces of code should also be made durable. how would you approach it? Because it's code that was generated by the agent kind of on the fly. Yeah, I think at that end is that we have engineers, we have developer principle around this. It's called test harness, right? And you have outcomes that you expect. You probably give it like a ratio of, you know, like what is acceptable, what is not. And potentially the... the super agent in that case, how you mentioned it, is probably just have to operate based on that boundary. And as far as I can tell, I still see some stuff, like someone shares with me Twitter, say someone doing some crazy stuff, like an agent running for eight hours and then did some things. The other thing is that if you don't have a test harness, if you don't have something to guarantee quality, like you rarely see any of those actually go out into production, right? You can just spend a bunch of tokens and doing a bunch of things, sure. But is it actually going to be served or are you just wasting, you know, electricity? So to that extent is that it's just that there is all this thing built, you know, as knowledge base in the industry that we can, you know, utilize, probably have to tweak for agents because of that. But to that extent, I would still approach it in a way that is like humans are also fundamentally unpredictable. Like how a junior behaves, how a mid-level behaves, how a senior behaves, how the staff behaves is entirely different, right? But you have companies that have engineering process to build in to make sure you kind of guarantee some outcomes. So what I imagine could happen, I don't know if this actually really happened yet, but could move towards is that now you're encoding that process to make sure the agents behave. within that boundary. And then to that extent, you'll probably need to get some contained, at least predictable outcome, and you'll probably tune from there. Nice. Nice. Okay, a little bit related to that, I guess. Before our call, you mentioned that you built internal agents for tree aging users and other things. Can you talk a bit about how you approach designing these agents? What are... the patterns that you see yourself going back to and what patterns do you think are promising and also not promising? Yeah, I think one thing we observe more now and I think there's also more realization is that CLI tools and APIs are actually great, right? And funny enough, CLIs has existed forever pretty much. If you look about Unix, it's like everything's a CLI. And APIs has become, a lot of that push has went around the 2010s, but it kind of faded out way. And it's just like agents reviving this, right, basically now because it's not humans, but it's the agents. And you kind of think of how to optimize the agents' time to do things. So on some of our agents internally, specifically, we have something called triage agent. This is, what it does is that Let's say, you know, a customer file bug report and based on certain behaviors, you know, on support, we have a linear issue created and there's a linear bot or it's more just a bot interface that is backed by an ingest app. So what it does is it's essentially sending events, like linear events, and then that can normalize into instructions to agents running inside our production infrastructure. And that thing will take down the code base or clone the code base and scan through it like how you would do with codex or cloud locally with your code with based on certain prompt instructions to check on this. Like, hey, this is the observation. These are the logs. These are things that have been observed. Like, give me probable causes through all these code bases. Because I think we have one, we have monorepos, but we also have like a bunch of repos distributed as well. So it kind of like, if something goes wrong, sometimes you might have to like look through multiple repos to make sure you get the full picture. And that has, that is actually designed a little bit around how human operates, it's like how we engineers operates, because the intention of that agent specifically is to reduce our, you know, time to like the triage time it takes sometimes because you're looking to login systems pulling different information that can take hours just to get a full picture and agents can do that significantly faster if you give it the tools that it needs so and yeah i guess it's like but to to one of the questions like the patterns um is cli is great Anything that is local is great. Anything that you can actually just spend time doing it locally as much as it can. So, you know, prepare a file system for it to operate on, the code that it can inspect. And certain things like, you know, the GitHub CLI client, so you can actually pull the reference points for you. And so that's actually where we're focusing on because from based on that standpoint, it's just that we operate more on that way ourselves. And so we encode that process into the agent. And most of our stuff is actually doable just with pure REST APIs and CLIs. It's kind of rare right now for some more agents to use MCPs. It does exist. It's just if you really know the schema you're working with, why spend time on it? And there's also the back and forth that it takes for MCPs to discover things in terms of like, hey, what are your capabilities? if you as the human curated beforehand, say, these are the capabilities I'm going to give you, and these are the intentions I need, then you instead of having to search through, you know, 100 plus APIs, you're just like looking at four. Yeah. Yeah. So that's like a lot more efficient in that way. So we're definitely looking, we're definitely like utilizing things like, you know, skills.mds. like more patterns as observed over time, just trying things out, following the trend a little bit here and there and see what works, what doesn't, and it kind of tweaks as we go. Amazing. Just wondering, is this a custom harness that you guys built or is this running on Pi or Hermes or Cloud Code or something that exists already? It's actually an ingesf. It is actually ingesf because we were actually trying to dock through some of our capabilities, too. And there's certain things we figure out as we dock through this. It's like, how can we experiment this? Does it get better output if I use 5.5 high versus 4.8 opus high? Or how do I switch between if I want to? There's, it makes us give us better, you know, understanding of how our customers will potentially use us. The other thing is just, you know, it's kind of proof of a point that like durable execution works. So, and we know ingest works, so it's just like, why not just do it? Exactly. I love that. I love that. Yeah, so it is actually ingest app and it's like, it's, and it runs as ingest functions. Nice. Nice. Very nice. I have a question, I guess, like, When you start creating one of these agents that you will run internally, especially the ones that you will graduate to production, maybe, do you start from like a fully agentic thing where the agent has all of the, just, you know, give it some tools, have it have, you know, access to some context and then figure it out? Or you restrict more and more into a specific set of workflows? Maybe like run books and things like that. Yeah, this one, I can't speak entirely for the team because I'll say like everyone's probably still finding their sweet spot. I have landed in something for myself just because I want the agent to work for me, not the other way around. And I'm the type of person that I really don't like babysitting. So one of the things that I do actually, even before agents were, we We spec out what we want based on, it's a document, it could be Notion, it could be unmarked down. In these cases, inside the repository, we have a planning file or plans and then we have each file is a scope to what it's supposed to do. And there's research done, there's context built in. The research itself is initiated with an agent. So you can think of it as, you know, I start a codex session or a cloud session, and then the first prompt would be like, okay, let's draft a plan for X, Y, and Z, or something like that. And if there's additional context to give it, and then kind of see what it comes back with initially, and drop it into a file. It could be a markdown. For my case, because I'm an Emacs user, I use org mode. So it's just easier to read. But that doesn't matter. It's still plain text, right? So the agent doesn't care. The main point of this is that I need to understand what this thing is going to try to do, and does it fit my intentions? Because I've been iterating with agents for at least over a year, and I was playing around, you know, right, and Sonic came out, and it's like, this thing is useless. Until Opus became, okay, this is actually kind of useful, and then GPT-514 becomes, okay, this is actually even more useful, right? Because it can actually read intentions better. it has better situation awareness about your, you know, a prompt that you get instead of just like taking it and run with it. So that part, but still, like, I need a document that can share between a human and an agent, a human being myself, and I will read through what it says. And usually when operating that way, it will actually have open questions and code it into the doc. And the other thing I will always do is I will ask it to break into phases. Because one thing is that the agent's gonna write the code, who's gonna review it, right? And if it's not reviewed, like some individuals on the internet do, then you just get a bunch of slough, that you don't have confidence on the code that's being shipped, right? So I need to make sure that the code is done in a way that is reviewable. you know, around maybe like 500 lines or 1,000 lines or something, it's like, that's like acceptable, like not a 10,000 line PR. And be able to review that, make sure that things like tests are added, behave like, you know, test, this is all standard engineering practices, but more accelerated by the agent than, you know, myself writing it by hand. And that actually has provided a lot of predictive outcomes. And this actually is a lot of ways that I will start nowadays to even starting a project. Actually, you know, having the agent built itself is that let's draft a plan because if you don't know what you want, the agent's obviously not gonna know what you want. So it can be like, there'll be times that we'll be spending hours, just like, you know, three hours just going back before the agents, I don't want this, I don't want this. And actually, you know what, this might be a good idea, add this. refine it, and have the agent start implementing it. But usually the specific instructions I'll give it in the case is like, commit in small logical chunks. So each commit itself is reviewable. So that way when you push us to PR, I can look at each comment and see what it's doing. And there's something like small tweaks in there that it's like kind of works for myself. And usually that's at least, I know some folks in the team has kind of picked up what I'm doing and then I start modifying it for themselves. But that has been working pretty well. Like there are internal projects actually built entirely using that flow. And we can ship it because we have confidence in it. Amazing, I like that. And now I want to go a little bit into MCP and I know Ingest has a developer server that's shipped. MCP server out of the box. It has a bunch of tools that lets cloud code and cursor and friends release functions and events, monitor runs, et cetera. What's the developer experience you guys were chasing when you build an MCP server? And what does, in your opinion, an AI assisted debugging session should look like? Yeah, I'll credit this actually to one of our teammates called Jacob for building this out. But more of the things is, I'll say the intention behind this was mainly making sure that we follow, you know, like what the industry's moving towards to and making it easier to develop interest functions. And one of the supporting factors of this is obviously documentation, right? surprisingly reads a lot of documentation. So we actually, we are surprisingly having more documentations that we previously had because we needed to be able to reference things. But the other thing is exposing APIs through MCP so we can understand intentions and understand what it's capable of. So the MCP was added to mainly really to help people adopt ingest in a more friendly way with like reducing friction because that's kind of what we always try to focus on is just reduce friction and then like the entry point of um ingest is also you know not an exception for for it uh as well yeah it's yeah sorry i'm probably this there's nothing i can add on but that's that's roughly it really no problem so what do you personally think about mcpe as a standard do you like it do you don't like it do you think it's interesting um to have it as you know, as a way to consume tools for your agents or as a way to expose your own systems to agents? You know, I'll give you a very non-deterministic answer. It depends. Like, it's, I guess it's like take a step back on, you know, just like understanding protocols in general, right? We used to have SOF as a communication protocol. We have RPC. We have HTTP. We have GRPC now. And some of the API protocols, you have REST, you have GraphQL, right? All sorts of things. You have different types of operating systems like Windows and Linux. So personally, do I like it? I'm kind of neutral. To the extent, I view it as one of the options available. for an agent to communicate just like how the browser would decide to communicate and how like, you know, what people you decide to use for their operating system versus like Linux, Mac or, you know, FreeBSD or whatever it is. And it's more really just like, is this gonna stand over, you know, time, like last over time in that sense. There's nothing really need to battle test about this because it's nothing it's gonna put to a high load. It's just an HTTP server. We know how to optimize that kind of stuff. But it's really just that, is this enough? Or do we need certain cases? And MCP will fit, will be a great fit for certain things. Like, for example, I personally use Fastmail as my email provider for personal stuff, for various reasons. They have an MCP server, which makes sense because it's like, I can't imagine Fastmail providing a CLI to use. And there are other cases like Datadog, Grafana, those are MCP servers. Because like, what would you imagine a Datadog CLI look like? Or a Grafana CLI look like? So there are cases that it just fits better. Sometimes it might be less optimal in terms of, you know, like token consumption and such, but everything has, you know, a sweet spot. I imagine that MCP will continue to exist for what it's meant to because as far as I understand, it was designed to make sure it can connect various things that previously is a little hard to connect or for the agent to figure it out. So for that sense, I think it accomplishes goal. Will something come over and take over space? Maybe, because we engineers like to replace things every couple of years. All right. let's pretend you take a you know a year or a year and a half break and you know the agentic ecosystem is evolving things are changing now we have like you know super massive intelligent models and all of that um what does you know the arrival of super intelligence ai um excite you about or do you not care at all about this um if there's anything um that I think I will like about it is like if I can give, if it can give me back time to do other things, that'll be, that'll probably one day I will approach it because like I have an 18 months old and you know, playing with kids is fun. And then it's like, and then, and the funny thing is when, you know, Asian makes coding so much faster, right? But it turns out coding was never the bottleneck. I'm actually spending more time working than I previously was. Because now, because the other parts of the, you know, the architecting, the designing of the systems, understanding constraints, you know, like making sure Asian is doing their job. It's kind of like how you manage teams of engineers. You're managing like AIs. And if, you know, something can read my intention well and be able to get it all the way to the end and, you know, accept and fits the quality. that I accept, then that means that I just get time to do other things that I probably might not have time to have done. So if there's anything, I'll probably just like, you know, treat it that way. I'll say like, I know, I wonder if, I'm not sure if you have some intentions about or asking about like job security and such. It's like, for me, I personally don't care. But... I know that there are a lot of people who are a little worried about, you know, what their skills mean. And there's, the way I'll put it in that sense is that software engineering has never ever had been about coding. It has always been about understanding constraints, understanding your situation and choosing the best optimal solution, right? And coding has been the last final piece of encoding that decision into bits and bytecode and such. So in that sense, if you look at it that way, then really, AI should have made your job a lot easier to do. Or probably what I could think about in more of an optimistic way, in that it's like, what else can my life improve because of this? And funny enough, I probably think that because of all these agentic stuff going on, I think there'll be, I suspect there will be more people interested in actual things that are human touch, like a painting or certain things that are made by hand or et cetera. I assume that it's like premiums of those will go off a little bit. I don't know, I'm not trying to break the economy obviously, but more so that you can have too much of something. people want for something decimal scars, right? Yeah, that makes sense. Is there anything that I didn't ask you that you wish I did? Nothing specific I can think of. Oh, yeah, I mean, obviously, no. And if whoever's watching this, please try our ingest as your next agent workflow. And please get our feedback. The ingest dev server is open source. And thanks to AI, some people are actually submitting PRs to help fix smaller things. But if you want to understand how it works, the code is there. So play with it, it doesn't require a sign up. Obviously, I'll be happy if we have more people sign up. But I would like this to be a little bit more less about how I make agents work and focusing on making agents work for what you want. Nice. I will put all of the links in the description below. And where can people find you? I have a website called like dwu.sh. Lincoln is probably one of those. I'm obviously on GitHub a lot. So, you know, if you want to shoot out an email, it's probably like listed somewhere around there to reach out if you like. Cool. Amazing, man. Yeah. Thank you. Thank you. And thanks for your time.

Darwin Wu is a tech lead at Inngest, an engine for durable execution — the thing that makes a workflow keep its promises even when an LLM call gets cut off midway. We sat down to talk about why "agents are workflows" is more than a slogan, and what it actually takes to run them reliably at billions of executions a month.

The through-line: agents aren't a new failure mode so much as an old one wearing new clothes. You've got non-deterministic behavior layered on top of every distributed-systems problem you already had. Durable execution is how you put a floor under that — and Darwin's case is that the floor belongs in your code (a single `step.run` that collapses queues, retries, and state), not in some separate piece of infrastructure you bolt on and babysit.

In this episode, we cover:

- What "durable" actually means — and which work is worth persisting vs. dropping
- Why Inngest puts durability in the code instead of the infrastructure layer
- step.run(): collapsing queues, retries, identity, and state into one call without leaking complexity
- Scaling from millions to billions of executions a month, and where the real bottlenecks live
- "Agents are workflows" — why the agentic era is mostly a distributed-systems problem
- Human-in-the-loop via "wait for events": an API built for CI/CD approvals that turned out to fit agents perfectly
- Why agent failure modes are ~80% the same as traditional apps (it's still humans encoding intent)
- The triage agent Inngest runs on itself — in production, on Inngest functions
- Why CLIs and plain APIs often beat MCP for agents you control, and when MCP is the right call
- Darwin's own build-with-agents workflow: plan first, phase the work, keep commits reviewable
- A neutral take on MCP as a protocol — one option among many, judged on fit
- Superintelligence, job security, and why "coding was never the bottleneck"

⏰ TIMESTAMPS
00:00 – Introduction & Darwin's background
01:13 – What durable execution is and why it matters
04:27 – Inngest's philosophy: durability in code, not infrastructure
09:09 – Scaling to millions (and billions) of executions
11:08 – Agents are workflows: reliability in the agentic era
15:30 – Human-in-the-loop: pausing and resuming agent workflows
20:39 – Agent failure modes vs traditional app failures
22:42 – Building internal agents: the triage agent deep dive
29:45 – Darwin's personal workflow for building with AI agents
35:40 – Inngest's MCP server and AI-assisted developer experience
37:25 – MCP as a protocol: neutral takes and real use cases
40:27 – Superintelligence, job security, and the future of engineering

🔗 LINKS & RESOURCES

About Inngest
- Inngest: https://www.inngest.com
- Inngest on GitHub (dev server + MCP server, open source): https://github.com/inngest/inngest
- Inngest blog: https://www.inngest.com/blog
- LinkedIn: https://www.linkedin.com/company/inngest-inc/
- X: https://x.com/inngest
- Bluesky: https://bsky.app/profile/inngest.com

Referenced in this episode
- LangChain: https://www.langchain.com
- Spinnaker (deploy pipelines with manual approval gates): https://spinnaker.io

About the guest
Darwin Wu, Tech Lead at Inngest
- Website: https://dwu.sh
- LinkedIn: https://www.linkedin.com/in/darwin67/
- Bluesky: https://bsky.app/profile/darwindwu.com

🚀 Try Arcade: https://account.arcade.dev/register?utm_source=youtube&utm_medium=description&utm_campaign=external_interview&utm_content=inngest_durable_execution

Looking for the fastest way to build MCP servers?
https://docs.arcade.dev/en/home/custom-mcp-server-quickstart?utm_source=youtube&utm_medium=description&utm_campaign=external_interview&utm_content=inngest_durable_execution

💬 Join the conversation: https://discord.gg/GUZEMpEZ9p
🔧 Build with Arcade: https://arcade.dev
📖 Arcade Docs: https://docs.arcade.dev

#Inngest #DurableExecution #AIAgents #AgenticWorkflows #DeveloperTools #MCP #ArcadeDev