2026.02.04 Oluwafemi Shobande - Durable Execution

All right. Thank you. Good evening again. First time at Dev Club, so it's nice to see all of your faces. Just to kind of give me a fair idea of the demography that we have over here. I'm an engineer. Can you just wave? Software engineers. Okay, okay. It's like 95, 99 percent. Not an engineer. Okay, okay. Nice. So I can speak a lot of jargon then. Thank you. All right, so first, anybody here heard about this term, durable execution, before? If you have, just like wave. Okay, about three people, four, five. Awesome, awesome. It's a fancy term, I must admit, but my goal today is just to demystify it and explain exactly what's this. And more importantly, I built an open source, durable execution platform. And I'm going to be showing some of the unique approaches that I take to solving this problem of durable execution. So let's just take a step back, okay? And extend a bunch of the problems that we typically face when trying to build distributed systems and see how durable execution platforms help us solve some of these problems. And to do that, Suppose I have, I'm trying to build a little e-commerce website, I don't know, to sell some JetBrains swags or whatever. And a few steps. I want people to create an order, reserve some inventory for, say, 30 minutes. And after 30 minutes, if I haven't received the payment from them, I would simply just, like... release the inventory that was previously reserved so somebody else can order it. But if we receive the payment within 30 minutes, then obviously we go ahead to fulfill the order and deliver it to the customer. So keep that in your mind as this is a scenario that we're painting that would lead us through this discovery phase of why durable execution platforms are really good. So this is pretty simple, but how would we typically implement this today? I have a sort of pseudocode. It's mostly in TypeScript, but I think the code is simple enough for everybody to kind of read and understand the gist about it. This is just really treating the words that I previously talked about. Can you see the text from the back? Is it clear? Can you wave if you can? Awesome. Awesome. So the same thing create the order, reserve the inventory, wait for the payment and do something else. But anybody sees anything wrong with this code? No try catch. No try catch obviously. Awesome. One of the most important things here is how are you able to represent the weights? how would you typically do that? Can you simply tell your program to wait for an external event and pause? It's a little hard to represent that kind of concept in today's programming. And I'm going to be exploring the ways we would typically implement this today. So let's start building this step by step. First of all, we'll probably have two different endpoints. One endpoint for creating the order. And in that step, it's just going to be creating the order and reserve an event, as I previously mentioned. And we return a response to the user. And we have a different end point for the payment webhook or something just to receive a message when the payment has actually been made. And then we go ahead to update the order's status and actually fulfill the order. But something is missing over here. We totally botched the timeout thingy because now we have a bunch of inventory that was reserved but what if the user never paid within 30 minutes or the user decided to pay one day later you know we've reserved a bunch of inventory that nobody else could buy and we didn't really detect it so what would we typically do about that um the next thing is well let's create a cron or some background process that would pull our database look for unpaid orders whose creator that is like within the last 30 minutes right and then find those orders run through all of them mark release the inventory and mark the orders expired so this is one of the ways i would typically do this we we kind of have a different process handling of that. But there's another problem, right? So we solved the inventory reservation problem. As somebody mentioned, there was no try-catch. Things can fail. So imagine somebody made the payment, we received the webhook, we updated the order status, but our server crashed before we could go ahead to fulfill the order. So what exactly do we do in that particular case? just depend on vibes and hope for the best or is a failure of a first-class citizen in our workflow. So the naive approach would be obviously doing a try-catch, but again, that really doesn't prevent some kind of issues like what if your server literally fries or crashes or Kubernetes is restarting a pod or somebody just does something, like power goes up in your data center, you know? If anything terrible can happen, it will happen eventually. And we need to prepare ourselves for such eventualities. One way that we would typically go around this is to implement something called the Outbox Pattern. Anybody here? Have you heard of this pattern before? Okay, okay, cool. A few people, awesome. So the Outbox Pattern is a way to solve the distributed commits problem. Just taking a step back to the previous slide, the big issue here is that we want updating of the order status and fulfilling of the order to be an atomic operation. So we want those two things to happen or neither of them happen. But the issue is that because they are potentially different services, maybe different microservices, it's kind of hard for you to create a single database transaction that spans multiple. different services. So the outbox is one way to get around this and the story of the outbox is that you have a different table in your database called an outbox. It's kind of like the anti-feasus of an inbox. So the outbox is where you just put some mails that need to be sent out and then you have a different background process that would periodically go to the outbox, pick up those letters that you want to send and finally send it out. But the beautiful thing about the outbox is that now you can wrap both the updating of the order status as well as inserting into the outbox in one single transaction. So those two things happen atomically and then you can have a different worker that just picks up the messages from the outbox and fires it finally. goes ahead to fulfill the order. Any questions so far? Or things are clear? Basically, this is a queue. Yeah, exactly. Outbox is basically a queue in your database. That's all it is. And just some little pseudocode on what this would look like. So instead of just updating the status, we would create a single transaction that would update the order status as well as insert something into our Outbox table. And then we, again, another background process, which would periodically pull the outbox table, find all those pending messages in the cube, go ahead to fulfill the order, and finally mark the message in the outbox as this has been handled. So this helps with situations where either server crashes, at least you know that. the transaction was never committed or even if the transaction was committed eventually the worker would pick up something from the outbox and process it to finally fulfill the order. Well another option is well just use a regular queue instead of you know building one into your database and this is again a viable solution because the queue although it's you having to install a new infrastructure in your system that maybe makes things a little more complex instead of just relying on your database. But it is a viable solution. And it comes with a bunch of really good things. Like once the message has been delivered to the queue, you know that until it has been handled by some consumer, it's never going to leave the queue because it's just keep on re-delivering it over and over again until you finally acknowledge that this message has been handled. So that's a good alternative. But the other line principle is still similar. You still need to have a different process that listens for messages on that particular queue, consumes them, and does some operation. So it's more or less similar to the outbox, just a different mechanism altogether. Don't worry too much about this detail. It's just a pseudocode for the queue. So where do we start from? We started from we want to create an order. Reserve inventory, wait for 30 minutes. If payment arrived, do one thing. If payment did not arrive, do another thing. Now we have essentially split this logic into multiple different background processes, each doing a tiny portion of this process. And potentially this logic is maybe spread across different microservices, spread across different teams, maybe. Eventually, it's kind of hard for you to wrap your head around the process end-to-end because maybe your team handles a really tiny portion and you just emit an event and another team handles. Besides, debugging can be a challenge because now you have to figure out the correlation across these different services end-to-end. It is a challenge. Another issue that we haven't even touched on is potential risk conditions between all these concurrent processes that are operating on this single order. So imagine you had a situation where the payment was received at exactly the same time when the inventory was supposed to be expired or released back. So I kind of traced this scenario over here where the payments were received, but at the same time we're almost trying to release inventory, we need to think about what the concurrency model is, you know, and either you need to do some pessimistic, optimistic locking so that things don't get weird. But the fundamental thing I'm driving at over here is we're building more complexity over and over again just to solve a very simple process end-to-end. So this is where we are today. We have multiple extent points. We have an outbox or a queue. We have some workers. We have some pullers. We need to do optimistic or pessimistic locking and not talk about potential try-catches plus maybe automatic retries for different retry policies like exponential, etc. The code just ends up becoming pretty, pretty large. Yeah. This is enterprise software. But what he does a better way. And there is a better way. This is where durable execution comes in. It's a whole different paradigm for building applications where you kind of write what you want, the business logic, and the platform just handles a lot of these shenanigans that you typically would have to implement yourself. So things like waiting for events, going to sleep, In fact, with durable execution platforms, it's possible for you to, say, sleep for one month. It's perfectly normal. And I would explain how a bunch of these things work. And retries, they come out of the box. You don't have to do much about retries. It's just configuration. So I bring you iKey. iKey is what I've created. It's a durable execution platform, fully open source. There currently there's only a TypeScript SDK, but a bunch of other language SDKs are on the way. Maybe just if anybody's interested, a little lore about why I created this and why I named it this. Well, Ike means workflow in Hausa. Hausa is a language that's popularly spoken in Nigeria where I come from. So I was looking for really cool translations of the word workflow. Aiki sounded really, really cool. And then I found that there is Japanese Aikido, which is like a martial arts discipline where you do not fight your opponents with a lot of stress. You basically wait for them to attack you and you kind of redirect the energy in a super easy manner. And it really made sense in the model of Aiki because I'm... I'm presenting a platform where you don't really have to fight failures. You embrace them. Failures are a first-class citizen on the platform. And you kind of just go with the flow and write your business logic. And the platform just helps redirect all that negative energy back to wherever I came from. OK? All right. So remember where we started from. I'm going to be coming back to this over and over again. This is our North Star. OK? This is what we're trying to model. With iKey, iKey is made up of what I call workflows. So a workflow is, let me take this back so I don't distract you just yet. So iKey is made up of workflows. A workflow is, you can think of it as a virtual execution, a virtual thread of execution. And in the case of our order, instead of thinking of individual, like workup polars, or Outbox poll, as we can think of this order end to end as one single workflow. And that workflow means it starts with at the point when they create the order and it ends at the point when the order has been fulfilled. And we can kind of model this idea of business logic inside code instead of spreading it across many different pieces. And I will go into the details of this. These workflows are made up of tasks. So the tasks are like tiny execution units in a single workflow. These two top-level concepts kind of form the baseline upon which IKEE is built. So knowledge is going to concrete details, right, instead of building castles in the air. IKEE exposes a bunch of primitives, like tasks, obviously. So you basically just import a task. And the task doesn't really do much. It's just a wrap around functions. And the task has a name and it has a handler. The handler is just a function. It's no different. Excuse me. And the handler is all will contain the logic for that particular task. So in this case, all the functions that we had previously for creating an order, reserving inventory, releasing inventory, we just need to wrap them around IKEE tasks, first of all. And then we need to create a workflow. So the workflow is what glues all these tasks together. And I'm just going to give you a minute to read through so that attention is not split. You can just say yes when you're done. Oh, I can give you 30 seconds. Great. So what you might notice is this looks pretty much like the pseudocode we wrote. It's almost identical. The only thing it has is it's been wrapped around the workflow and those individual functions have been wrapped into tasks. The beauty of this is that, again, now you don't need a separate polar for figuring out if the inventory's expired. You don't need a separate outbox, you don't need all of the logic end-to-end for a single workflow is represented in here. And we'll be going into the details of what makes these things possible. First of all, before I dive into that, anybody here familiar with the concept of an event loop? I think most people should, right? Okay, cool. But if you aren't, so the event loop is one of those patterns that makes tools like Node.js possible. It allows you sort of fame concurrency in single threaded platforms. And at least the way it works in Node.js is when you have a bunch of blocking operations, maybe you're calling an API or reading from disk or performing some operation that's going to take a really long time. Node.js actually doesn't wait for that thing to happen. I think this is the same thing that happens in task programming in C-Shop and any other async and wait-based programming model. But essentially what it does is it just submits the I.O. operation into the operating system. And so in this example, we have two clients. Our application just has a single thread. There are two clients that come ahead and execute two different API calls. One is to read a bunch of users and the other is to get a bunch of orders. Obviously, this information is in our database, so we need to issue an I.O. call, which may take a few milliseconds or more. And that single thread doesn't handle client A, wait for the results of the database before handling client B. pushes the request onto the operating system, or some other lower level worker pull, which will take the actual operation, execute it, and whenever the result is ready, it would come back and add it to a queue that the event loop, which is basically the main thread, can go ahead and pick up from where it left. So this is kind of like the idea that gives the illusion of concurrency in single-threaded programs. And iKee works very, very similarly. Going back to the previous slide, when you call wait for an event, it technically doesn't really wait. What it does is it just traps into the lower level mechanism, lower level runtime of iKee, and it would save the state of the workflow. Then just go ahead. The thread is free to go ahead to do anything else. Either serve client requests or run some other workflow or do anything. And at the time when, you know, the 30 minutes elapses, the workflow comes back alive and continues from where it stopped. So it doesn't start from the beginning. It actually continues from where it stopped just the way it would typically happen in your regular traditional async I.O. So you can think of IKEE as like a distributed event loop. It takes the idea of event loop and goes ahead to extend this idea into distributed systems and says that, well, we don't necessarily need to store the state of a running program in memory. We can store the state of a running program in some durable platform like a database such that at some point in the future, the program can come back alive. and resume from where it stopped. I'm going to explain how the resumability works. But just one thing, it's entirely fine and normal for you to do wait 30 days in, durable execution. In fact, you can do wait for 100 years. Nothing stops you from doing that. It's not going to consume resources while it's waiting. You can have 100,000 workflows waiting. That's perfectly fine. Resources are not going to consume while they're waiting. I said I was going to explain how the resumption works because how is it possible that when you call wait for an event, it would come back and continue from the next line of code at a later time. The way this works is a mechanism called replay. You may be familiar with this from, say, if you've done any sort of event sourcing in the past, but The idea behind Replay is that when the very first time the workflow starts, it keeps an append-only log of the different steps or tasks that it executes. So the very first time, it might do task A, it's just going to store the results of that particular task in some database. And when you call wait, it would, you know, trap back into the iKey runtime which would release the worker and when the 30 minutes is up it would resume but this time the way it resumes is it takes the log of actions that it previously had on the first run and uses it as a reference point to kind of rebuild the state of the program so that it gets back to the point where it can continue from. So in this case It's literally just like running the function again. The only thing is that those tasks which were previously computed would be skipped entirely because it has the results of those tasks. And this is the source behind most durable execution platforms is this idea of we can replay things using the event log as source of truth. Any questions so far? Or things are clear? If I move too fast, let me know. Good, good. Yeah, go ahead. So what if things... Say that again. What if you restarted on a different machine? Awesome, that's a great question. I do have an explanation of the full architecture closer to the end, but I'm just going to visit that a little earlier. The state of the workflow is stored in your database. It doesn't really matter if... you review the state on a different worker on a different pod as long as you know each of them has the pre-existing code it can literally just pull out pull the log of events and rebuild its state from even if it's on a different machine it doesn't really matter as long as that log of events can be pulled from the database then it can review the states and that's actually one of the beautiful things about about the crash recovery and durable execution platforms is you can see it on machines. It doesn't really matter. The state can be rebuilt anywhere else, on a different operating system, on a different cluster. Who cares? Yeah? Yeah. Yeah. Great question. I'm going to come to that in a moment. So can we just hold on to that thought? Awesome. Any questions before I move on? Any other ones? Great. So this is what I was explaining. On crash, recovery just happens automatically because, in fact, you don't necessarily need to do anything because a different worker would just figure out that this workflow had crashed and would automatically pick it up, pull the previous log of events, rebuild the state, and continue from the next point. You don't have to do anything. There's a few challenges with replay. The two main challenges are matching results. Matching results has to do with, well, if we have this log of events that has happened, and this speaks directly to the question that the gentleman at the back asked is, how can we be sure that the current state of the code matches against the log of events? results that we kept, especially when I had task A. Maybe I did task A, task B, but then I changed my code and I switched the order of task A and B, and now task B is done before task A. So we kind of need to figure out how can we match the tasks against what we have previously stored. So that's the first challenge, and we'll dig into that slightly more. And the second is determinism. So just a fancy term is It simply means that code should, if you run the code multiple times, it should, first of all, follow a very similar path of execution as well as produce the same results on replay. So things like generating random numbers, for instance, is non-deterministic because every time you run the code it's it's a very different result or if you do date.now trying to get the current date it's very it's it's non-deterministic because you know the result changes for each run so these are the two classes of big problems that um we need to think about when doing replay um and uh let's let's just dive dive into each one of them and how ikey tackles tackles that So the first problem, result matching, Aiki does content addressing. Content addressing just means that you identify something by the content that was fed into it. So tasks are identified by the name of the task as well as the inputs, and we hash those things. So on replay, if... the task, on replay we rehash the name as well as the input and if it's the same we can just reference the previous result from that event log and see well I have a result for this particular hash and I just fetch that exact same result and go ahead to say well I have the result for this and the beautiful thing about this is if I wrote two different tasks that I was supposed to send emails to the exact same person it's more or less like automatically idempotent because the input is the exact same task and the input is the same. So that's the first thing that IKEEE does regarding results matching is content addressing. And it's a little different from some other platforms that you might have used which do things like positional based matching, but that's outside the scope of this talk. Five minutes, okay? You want me to take a break? Oh, I have five minutes left. Okay, okay, okay, okay. So I need to speed up. Great. One of the beautiful advantages of content addressing is, and this answers your question, is you can kind of add tasks or you can remove them as long as the hash is the same. you can always fetch the results of those previously computed tasks. And that's perfectly fine. And on replay, if you have written some non-deterministic code and the input of the task is different, obviously the hashing is different and it would be forced to re-execute itself. However, you can kind of model, IKU provides some mechanisms that allows you, like say even if the task is different, I still want this to not re-execute and it's like providing your own idempotency keys, that's available. And this is what I explained over here. So imagine you have two different tasks, the exact same input, but I want it to be executed twice. I can just provide my own reference ID or you might call it idempotency key to just be different and then to force it to execute twice. Does that answer your question? Does he answer your question? Okay. Another thing that IKI provides is the idea of conflicts. With content addressing, you might have situations where on replay the input is different just because I wrote some non-deterministic code. In that case, IKI has an optional opt-in model for determinism. It's where you tell it, well, if you realize that things are different, then do one thing on the other. And there are two major policies that are provided. One is an error and the other is like return the existing results. So what this does is, you know, imagine I had a task that I was supposed to charge a customer. And I know that it could be replayed, but I do not want to be executed multiple times. So what I can do is just provide a single reference ID and say that, well, if somebody changed the inputs of the particular task through an error, and I would provide that as my conflict policy. But if it's some other peripheral thing, which I didn't really care if it was re-executed, I could just say, well, return the existing result. And this is another way that this provides some sort of protection. Excuse me, charge is evil. It's a task. Oh, it's just a builder for adding options onto it. Okay, got it? Awesome. If there's one thing that you take away today, is that if you're using any sort of durable execution platform, the golden rule is always write your non-deterministic code inside tasks, not in the workflow code itself. The reason for doing them in task is the result is always stored. the outputs of the task. So on replay, it will just reuse the previously computed results instead of recomputing again. So for instance, if you want to generate a random number, do not do it inside the workflow, just do it inside a task, and then the results will be stored in database, and it will be replayed, and use the exact same results on replay. Okay, I think this is my final bringing it all together slide. This is, On a high level, all the different moving parts from IKEE, it comes with an SDK for creating workflows and creating tasks and creating some of these workers. There is a server. The server doesn't actually run your code. Your code always runs in your infrastructure. The server is just there for orchestrating. When you tell a worker to go to sleep, for instance, it's the server that kind of pokes the workflow when it's time to wake up. And it does that by sending a message into your, into a message queue which your workers in your infrastructure are listening on and then it would wake up, go ahead, find the code path for that workflow, rebuild the state and continue from wherever it's stopped. And you can horizontally scale your workers as much as you wish. It doesn't really, you know, that's entirely up to you. Yeah, there's some interesting points around how the concurrency model is, but I think I'll skip that for Q&A if anybody's interested in that. Thank you. I think this is a good summary of what this platform's provide. Yeah. Do we have any more questions? Oh, you can answer. That's why I reserved time. Yes. I have a question. Yeah. How do you compute those panels? So let's imagine you want to wait for... I know 45 minutes, in 15 minutes the cluster crashes and then summer time change comes in. So I'm interested in this. How does this... So the... I guess you're speaking about the issue of clock skew, right? So the timeouts themselves are... Let me go back to the architecture. When... You write a workflow that says wait for 45 minutes and things crash. At the point when it says wait, what it does is it sends the information about this wait and when it's supposed to wake up to the server. So the server persists that information about this workflow saying that, well, this is the point in future when you are supposed to wake up. In which form? In which form? Is it like Unix timestamps? Yeah, just timestamps. Yeah. Does that answer your question? I mean, yeah, this will survive. Yeah. Other questions? Yeah. Where do you write a different repository? How do you place the worker code in worker and some endpoints on the IT server? So IT server never runs your code. Your code is always running in your infrastructure. So think of it as it's kind of like you write functions in your code. But the functions don't run just yet. It runs at a later time when you call it. So the way you would do that, so let me show you an example over here. So imagine that previous workflow which we had created for processing the order. So the way it would happen is we would take, at the time when the user received, when we received the creation, order creation, we would just start the workflow. So process order, that's the name of the workflow. Oops. Here, that's the name of the workflow. We just start it. And this is like a reference to the actual function. It's like you calling a function, essentially. But it's not starting immediately. It's starting at some later time on potentially a different worker. So your workers themselves are part of your codebase. They have access to those functions. Does that answer your question? How do you specialize your work out to what? To your needs? Maybe give an example. How do you call specific endpoints in your infrastructure, for example? Not necessarily doable, for example. So the workflows, the workouts are something that you never have to think about. When you run a function and you deploy it on the Lambda or whatever, the Lambda itself is kind of like what I call a worker. You never have to really think about it. What you always have to think about is your actual implementation details of your function. And your IQ makes no reservation about what you can write. You can write, call whatever service is. It's just regular TypeScript code or any other code. It can do anything like call some other microservice, send monitoring metrics to your platform. It's literally just regular functions. That's all it is. Yeah. Yeah. Final question. Okay. So let's imagine we use IK for bank payment and one day clients get minuses from their accounts. Is it ever possible not to blame IK? Well, one thing I can tell you that IKI provides is very good debugging tools that kind of gives you a full view of exactly all the different steps that your workflow went through. So under the hood, every workflow is a gigantic state machine. And as tasks start, what it does is it transitions the state of this machine. You get a user interface for free actually with iKey. I can just show quickly, one sec. You kind of get like a web dashboard for free with iKey where you can see in details every single step that the workflow went through and that probably helps you in figuring out exactly what happened to my workflows and you can see the exact point where things went south. But that probably would never happen. Yeah. These things never happen. So, thank you, Femi. All right, thank you.

Stop losing state when your services crash. 🛠️

In this session, Oluwafemi Shobande (Senior Software Engineer at Bolt) breaks down the power of Durable Execution—a paradigm shift for building long-running, fault-tolerant workflows that survive infrastructure failures without manual intervention.

Whether you’re dealing with distributed transactions or complex microservices orchestration, traditional "retry" logic isn't enough. Oluwafemi introduces Aiki, a new open-source platform designed to virtualize execution state, ensuring your code resumes exactly where it left off after a crash.

What you will learn:
- The core mechanics of Durable Execution and why it's crash-proof.
- How Aiki differs from traditional workflow engines.
- Building resilient, long-running processes in distributed systems.
- Live demo and architectural deep-dive of the Aiki platform.

🔗 Resources:
- Aiki on GitHub: http://github.com/aikirun/aiki
- Connect with Oluwafemi: https://www.linkedin.com/in/oluwafemi-shobande/

Subscribe: https://www.youtube.com/channel/UC9lRNfzenw_7MVmDPFyRFjQ?sub_confirmation=1
Our website: https://devclub.ee
Our blog: https://blog.devclub.ee
Facebook-group: https://www.facebook.com/groups/devclub.ee