Event-Driven Applications vs. Durable Execution in Practice • Stephan Ewen • Devoxx Poland 2024

Okay, hey. Good morning everyone. Thank you for coming to this first session of the day. Happy to see people here. My name is Steven. I want to tell you a little bit about durable execution today. I work on an open source project called Restate, but the talk is more generally about the idea of durable execution and How it yeah, how it can can help with building highly resilient applications and applications that can pretty much crash at any point in time pick up very well just where they left off and Also a bit of the difference between event-driven applications, you know We're building this with message queues systems like Kafka versus durable execution. What are the differences what I expect I'm just out of curiosity how many Folks in this room have actually heard about durable execution before and know the term. Five, six people. Oh, no, that's something. It's a somewhat, I don't want to say it's a terribly new paradigm. I think it's actually been around for a few years, but I don't think it has actually made it very much into the mainstream. But I'm thinking that is about to change. There's almost like a bit of an explosion of, I think, durable execution projects coming up at the moment. And yeah, so let me introduce it to the concept. So just just just very coarsely trying to to place it into context when So I'm going to talk about durable execution mostly in the context of of event-driven applications here so event-driven applications being That's not the perfect definition, but like let's say everything but with Kafka something where you write an event to a log and then you process it out of a log and it's actually a pretty broad spectrum. There's more the analytical side of those where you're doing streaming, streaming analytics, materialized use, and then the more transactional side where you're building state machines, active systems, event-driven workflows, and so on. The analytical ones being kind of what the OLAP-style databases do. large queries, joins, aggregations, transactional really being update different rows, update different states, keep consistency. I think this part of the spectrum has gotten a lot of attention over the last years, lots of big projects created around this. Apache Flink, Kafka Streams Materialise and so on, kind of serve these use cases. This is a bit of an underserved category, I think. And this is exactly where durable execution I think fits in extremely well trying like making it really easy to build things like event driven workflows for consistent state machines even like extra-style programming distributed messaging and I think that's a that's a big thing because I think that side is where a lot of the important stuff happens in applications that were building like processing of orders The new fancy stuff, AI engines fits very well in that category. Service orchestration, payment processing, almost everything that really changes the state of the world and mostly produces the events that then on this side here are actually analyzed. So not all of these use cases always have to be built in an event-driven way. But I think people often result to building them in an event-driven way because of the kind of reliability that many of those use cases want. If you click a checkout button in your app, you want the checkout to happen. So very often, this is actually done by putting an event in a durable queue that is then consumed by a workflow. So anything crashes, the event is still durable, and the checkout actually continues. This is a bit of a biased view of the world that might come a bit because of my background. I worked on Apache Flink for pretty much 10 years since we started it in 2014. So I do have very much this sort of view of the world that more and more of our applications do want to, or more and more people do want to build applications on event streams and we have this kind of split world where Building event-driven applications if you're doing something analytical something oil AP style works actually pretty well But if you want to build it in a transactional way, you're pretty much still You're pretty much still at the point where you kind of hand roll Applications with the Kafka consumer so if you're trying to yeah If you're trying to actually build on like an event-driven workflow very often you have a very thin layer of tools and mostly you just like create Kafka topics and and consumers and you know reason about offsets and write a database and so on. It's kind of a very like low-level low-level kind of thing and the thesis of this talk is a little bit that durable execution is actually an absolutely great tool to build all these types of all these types of applications anytime an event needs to do something update different systems in the world reliably often more than one system if you're talking about a workflow style use case. Okay, I do want to quickly mention so durable execution is maybe if you haven't heard the term durable execution before sometimes people also call it workflows as code it's kind of it's very related it's almost the same thing in many ways there's a bunch of systems out there these are probably not all of them i'm sure i forgot something that that implement this paradigm uh some have actually been around for quite a while i'm not sure if anybody recognizes that logo that's that that's azure durable function for unlike on the microsoft azure stack which actually implements that paradigm. I don't think it's an absolutely great user experience, so maybe that's why it hasn't really caught fire. Temporal has been around for a while. Reset, the project we're working on, is relatively new. There's others as well. They all have slightly different flavors in which they do it, mostly in how they, I think, what kind of package they expose to the... to the outside world, so durable functions. It's kind of really serverless orchestration functions, temporal in the shape of workflow, restate, one the shape of distributed RPC and event handlers. But the thing that they all share in common is the following idea. So I'm using mostly restate as the system I'm using in this talk, but most of the things I'm talking about are general that should apply to not just restate, but to more systems and implementations here. The one thing that's kind of the core idea behind durable execution that is shared by all of those systems is the idea of like a fast and efficient journaling of the execution. You can think of that as you have a fast log running in the background that persists intermediate steps of your code as it executes. So, okay, the animation was way too quick. Let's actually wait here for a second. Let's assume we have a chunk of code. I actually picked the wrong code. I picked the JavaScript version, then the Java version, wrong one for this conference, but bear with me. Okay, so if you look at this function here, it's basically doing three, four steps. What it tries to do is it creates a new role, assigns a set of permissions to a role, and then assigns the role to a user. You can almost think of that as it is kind of a workflow-ish thing. You have those multiple steps. You'd rather like all of them executed in the end, and you want them executed consistently in the sense of if we're creating a role, we're assigning a set of permissions, we're assigning this to a user. Assume all of those are like API calls to different services. a role service that we're calling to create a new role, assign the permissions, and then we have a user service that we're talking to to assign this to the user. Let's assume we want to avoid situations where the last step, we're applying the role to a user. We're crashing in the middle. And now what do we do? I mean, we can try and run the whole thing again. We might have actually applied the role to the user already. So we're creating a new role, setting a new set of permissions, applying a second role to the user. It's probably not what we want, right? Because if somebody then removes the first role, the user still retains all the permissions because we actually gave them two roles with that set of permissions. So it's kind of these small things that you can probably solve this. You can try and say, OK, I can try and rewrite this application that in that way to help it recover from that situation. But an even easier way is to use something like durable execution. Let me just start this again. So what you can assume Durable Execution does is every time you have these kind of steps here, in this case every time you see a context.run, what the Durable Execution system does is it basically places an event in a background journal that says I'm capturing the result of that statement. and recording that this has happened. So in this case, you know, we might have created the role, we get an ID for the role, now we're looping over our array of permissions and assigning the first permission that has succeeded. And let's assume now something happens and the whole thing goes down, goes up in flames. And then this thing is retried, but it actually finds, okay, I have two things in my journal already. So as I'm going through this, I'm actually taking the values from the journal and then I'm continuing from there. And that gives you a very interesting characteristic. That gives you the characteristic that each step, when it has been completed once, is guaranteed to never be re-executed. That's kind of what you would get if you try to rewrite this in a BPM and workflow style. But you get this basically in simple code and with basically control flow, like I can use for loops, I can use exceptions, I can use if-then statements, and so on. You still you still have to worry about the fact that each individual function here might be executed more than once because you might actually crash Just while it executes and while it's being written to the journal the result, but as soon as you've made it past this point you're sure It's never going to be re-executed and you can always be sure that the value you see here It's going to be the same value in every recovery and that actually makes a lot of things easier because now you you really have to just assume I'd important to which I'd impotency for each individual operation And the thing as a whole, you don't have to worry about the thing as a whole having to somehow work. And pretty much all durable execution systems do this by having something like a queue, a journal, a database running in the background that accepts these rights. The absolutely critical thing here is if you're using this if you're using this paradigm, you're adding a certain amount of overhead, of course, because you're now running the function, you're taking the result, you're persisting it somewhere. And so I think the thing that's being sort of sorted out and where the different implementations of durable executions diverge a little bit is how exactly they implement that. And how fast and lightweight can they make that specific operation, like that persisting of intermediate results? Because it can be very fast and very lightweight. You can use it in a lot of places. If it's something that actually introduces a significant latency, then you probably want to use it in some high-value cases where latency isn't absolutely critical, but you don't want to probably use it anywhere where it's in the synchronous interaction path. So the system we're working on, Reset, is actually one that's particularly optimized for the latency side of things. you know, and makes an effort to keep the overhead of that very low. But yeah, I think this is probably the core idea behind durable execution. And let me show you a little bit where this can help and mostly contrast it also with building things with, let's say, directly with Kafka instead. Okay, so the... The case for this talk is, yeah, durable execution is basically, I would say, the up and coming way to build transactional event-driven applications. And I'll try and motivate this with a few code examples and a few small demos here. OK, so let's try and do something very simple. Let's take the... Let's take a very simple piece of code We're not doing we're not doing anything fancy here. We're basically just saying okay what we we have a case where There's a certain update to our Certain like yeah in this case. It's not even updates like check check user is the service that's running in the background we do want to We do want to interact with that service. We're getting the events out of a queue like Kafka. They're produced by different servers. We're using Kafka to communicate between them. And this is basically all we're doing. For every event that is in the queue, we're trying to call this downstream service. So it's the simplest thing right now. We're only doing one step. And this is implemented in an extremely easy way. This is kind of a utility function here. It just creates a Kafka consumer, goes over every element here, and takes a consumer batch out of Kafka. And for every record, it processes the record, which basically is making an HTTP call to this service. So that's our starting point here. Let's actually run this and see what happens. And I hope you can actually see this from the contrast and everything, roughly, and from the font sizes. OK, this actually doesn't look great. We're getting a couple of events per second out of this. This is way beyond what usually you would assume Kafka or any service can do. Let's actually look just quickly, why do we get such a... low data right here. And if we would debug into our services, we would see that this CheckUserService here does a few lookups in the background, and it gives you a latency of average 80 to 100 milliseconds to respond to a call. And that is already a bit of a tricky thing. So we're actually taking events one by one. Kafka we're making a call you know everything takes maybe an average 100 milliseconds to respond so we're basically down to 10 events per second even though probably our entire infrastructure can do a lot more and just because we kind of have this mismatch between between like the message queue on the one side where you pick up events need to acknowledge them and the sort of synchronous interaction with the system downstream That's not yet a workflow style thing, but I think it's an interesting point to start motivating the idea of durable execution because the one thing it really is actually very good at is bridging synchronous and asynchronous worlds as well. So let's actually run this same thing through a durable execution system. In a durable execution system like Reset, the code looks a little different because you're typically just writing this like durable handlers on our case our durable handler is is here handle check event which which gets this event you know we're taking the key making making a call and and then that's it and there's we go back to the slides for one second one of the one of the ways that basically durable execution systems work pretty much pretty much all of them is Whenever you say, okay, I want to make this call here, you have something in the Durable Execution Engine that records that this will happen, and then it basically will make that invocation and retry that invocation for you, you know, keeping the journal. So one of the nice things is that you're basically inverting the responsibility of calling the service already. Instead of saying, You're running a Kafka consumer that supports to pull events from the broker in a durable execution system. You would have something more like you're having basically a queue of the recorded invocations that you're telling the system, the function executions that should happen. And then it actually pushes this to the target code. And so if we're running this, we should immediately see that this is Getting getting a very different sort of throughput characteristic just because we're actually switching from a pull to a push model implicitly I'm sure both of them up right, so What I basically need to do here is just like start piping the events into the system and we should see window yeah, okay, this flows actually much faster at because we're With basically we've basically switched from a from a pull to a push model here Maybe that's a very easy case, but I think that's maybe the first one to start here. All right. Let's go to a slightly more interesting one. Updating multiple systems prevent. This is going back to the example that I showed before. Let's assume we want to do that exact workflow, the authentication service where we create the roles, the permissions, and then the user. Let's assume we do actually want to write this with With something like like Kafka or so because it's absolutely not impossible to do this actually It's very very well possible, right you could do something like this Somebody says okay. Hey, I want to I wanted to go through this whole sequence of steps So the first thing I'm gonna do is and cue an event that describes hey, you know Create this create this set of permission profiles that role for that user then we have the first So the service that consumes from that queue creates the role, then puts an event in the queue that says, oh, basically that role is done. The next thing that you need to do is apply certain permissions and assign it to the user. Then that's the next service that consumes that, does that, writes it again to a queue. It's kind of possible. You've now sort of implemented a bit of a step-by-step sort of sequence of transformations, each with durability in between them by writing this to a message queue. This is actually good in the sense of it's going to be reliable. It's going to give you the same type of guarantees as the other implementation. Each individual step needs to be important, but other than that, this will actually get the job done. This will make sure that you don't get two roles assigned to a user, and whenever you kick something off, eventually that user will have that set of permissions. It's good in that regard. It's probably a little annoying to write it, like write to a queue, consume from a queue at every step. The thing that I think makes it really annoying is actually the observability part of it. Because let's assume we have a user called Knuth that for some reason didn't get their permissions. We wanted to upgrade them in the system. We didn't get the set of permissions. And now we have to find out why. And it basically boils down to going through all the queues and understanding, OK. Like where is our Knuth event stuck at the moment? If it's stuck in this queue, then it means, okay, we've actually, you know, those services did their job, but for some reason, our problem is here. We're not making progress beyond here. And I'm not sure how many folks with you have actually built these sort of, you know, microservices with, yeah, with queues and PubSub and so on. It's, I think the observability part is one of the ones that we tend to always struggle with. I started with the most. If you compare a little bit the stream or event processing approach with a durable execution approach, both of them don't do something very different in the end. Because we just mentioned, in the durable execution side, what you also do is, after each step, you write something that records the fact that that step has concluded and that you can rely on for the next step. One interesting way to think about this is that durable execution is sort of a bit of a... You can kind of think of this topology as like folded into itself. Instead of saying you have a few queues, you know, that you connect one after the other, you basically have one queue that you sort of like feed everything back into. I don't know if that makes a lot of sense, but one way I always thought about it is going back to the theoretical computer science class in university when the Professor first said, you know, whenever you have a while loop, you basically have a Turing-complete system. I think durable execution is kind of like a while loop over a queue that kind of interprets the events and the code for you. That has kind of two very interesting characteristics. The first one is what's really annoying here is the fact that you have to sort of beforehand agree. the sequence of steps it's almost like you're manually doing something like loop unrolling after write every step down and then put a queue in between and you know it's just a small sequence of steps that's fine if it's always the same sequence of steps it's fine and having this sort of more loop characteristic gives you the ability to also do things like you know for loop if then where every event might actually different different flow so The way this actually looks like in a durable execution system is you have calling the function, you're doing the steps, every step writes an event into the journal. The journal sort of gets reinserted to the queue, but you can think of it as it doesn't go into the queue as an individual event, it actually goes into the queue as an event that is like a sub-event of the original invocation event. So you kind of keep them together. Every event has like a virtual sort of... sub log or so of individual steps. And then when something fails, what basically happens is the same thing as in any other queue, you would actually push the event, but you would attach the previous journal because it's sort of enriched information to the event. So you re-deliver that event to the application, but with the context of how far you actually made it last time. I said a lot about observability here, so let's actually look at what that actually means here for our case. This is the implementation in a durable execution system. It pretty much, this is the Java version now, and I think without the loop because of simplicity here. This is basically what it looks like. I think it also looks much simpler. It's much easier to understand than if you fragment your logic and say, OK, I have different services with different queues and consumers. Basically, just writing a sequence of steps. And every time you want something durable, you have to attach it to the context. Let's actually do something slightly different when we're running through this. Let's actually do it. in the debugger and kill it at a certain point in time. Okay, so first of all, I actually need to pipe in a few events. Okay, we should see those going through right. These are the workflows, and each of those is like a full execution through the workflow. I actually have to hurry up before all my test data is through. So let's assume we're doing... We're doing this thing now where something is stuck. So I just simulated this by placing a breakpoint here. So some event doesn't, or some operation doesn't make progress. Okay, coni14 was the one that didn't make progress for us in this case here. So what can we do in such a system to figure out how... Where things are stuck and the important the interesting thing to go back is we don't have like all these different intermediate topics or so to search through we really have just one one cue that that has all the information and that allows us to do to do things like basically almost treat this cue like a like it like a database of ground truth and and do certain things like Ask our system. Hey, you know, do you have to have like what what do you know about the key Connie 14 here? I took this from the debugger, from the breakpoint. You can think of this. You can take this from the tracing system. And it tells you, OK, for coni14, I have an event in the queue, 51 seconds. It's still running, but it's been running for 51 seconds. Something might be up there. OK, let's actually look at what is up there. Let's ask the system to give us the context. the context information that we have for this specific event. Like the, I'm not sure that you can read this, but the different sort of like sub-stages, sub-events that it collected for that event. So we can see here, okay, there's side effect, create role template, completed side effect, apply permissions, completed, and it's running like after step two, which makes sense, which is exactly where we in the end placed our breakpoint. So those two steps concluded. The steps are going, let's actually start it. If we try this again now, the whole invocation is through. There shouldn't be anyone anymore for Connie. So it's actually nice in sense of observability. We've kind of gotten to a point where we have the reliability of distributed queues and distributed workflows, but we have almost like a central database or so that tracks this for us. And that's because we're... sort of the journal of our code being associated with the event actually has all the information we need. So another way to think about this is you kind of get out of this simple code the same sort of like observability as you otherwise get from a workflow engine where you can look at, okay, you know, like where is my workflow stuck and so on, what operation is it? But you get this from basically almost arbitrary code. All right, let's try something else. Putting back an unprocessable event. I'm not sure how many folks of you have struggled with this exact thing. It's been one of my personal highlights in the sense of this is the thing that I found takes almost most time to solve from all the patterns that we ever had to work with. The use case is the following. We're going to have an update event for a user that we want to put into a service, but the service downstream is actually eventually consistent in the sense of it might not yet have processed all previous updates. There's another interaction with that service going on. So at some point in time, the service might just say, OK, I actually can't process this update right now. Or maybe when you process the update, you have a precondition for a version where you need it to be, and it hasn't actually caught up with that version yet. So it might actually be that, you know, we're calling the service, when it tells us busy, we'll have to wait for two seconds and then we make that call again. It's a very naive way to implement this. And as you might guess, if we're running this now here, in the very naive way with Kafka, this is going to be absolutely brutal. It's just not going to work actually that way. So we're running a few events, then there's one that actually stalls because the downstream service is busy again. No chance. This is not what anybody would deploy to production. And it's a very typical characteristic, actually, of many sort of event-driven designs where you have this head-of-the-line waiting effect, right? Like you're taking events off one by one, putting them into a system, and as soon as you can't process that, what do you do? I think it's an especially tricky case with Kafka because you can't really acknowledge that event because then if you actually crash you've committed the offset and you don't get it back after recovery. But if you don't acknowledge it, if you don't commit the offset then you can't actually commit the offsets for any other event that comes after it, right? Because you're basically just moving the cursor position. So I think this specific case in itself is probably created in like in all the big companies that use Kafka that I've ever talked to projects like consumer proxies and so on that deal basically just with the sort of decoupling of hey here's your events coming and we're trying to send them somewhere where it takes a while where we might have to put them back dead letter queues taking events aside persisting them in a database putting them like in a delay deferred topic and all this I think this alone is I don't know has taken sprints or so to fix it because it's actually not that uncommon that this that this actually happened so one one interesting a thing we can do in a durable execution system. I'm actually just taking this out of the debugger for simplicity. And I like this again. OK. If we're doing this in a durable execution system, we basically have the ability to write it exactly like this. We're making a call to a system. We're saying, OK, if the user is busy, we basically sleep. Then we make this call again. And the interesting thing is, This works not just if we say, try it in two seconds, this would also work if we say, try it in two days or two weeks or two months. The interesting reason why that actually works is the following. Again, assume that every time a step like this happens, there's a journal entry that says, okay, this has happened. So this entry here has happened. The sleep is the same thing. You can see we're doing not just a standard sort of thread.sleep, we're doing a context.sleep, which actually means there's like a sleep entry in the journal which tells us, okay, I want to wait until that specific point. Maybe for two seconds we're not seeing too much of an impact, but if we would actually make this two hours... You'd actually you'd actually have the ability to say or you'd actually first have to think okay is this this thing still up after two hours probably There's a fair chance that in this two hours something happens, so it crashes. Okay, let's let's say after the crash we again replay You know we find this response in the journal. We certainly don't want to wait for two hours again, right? So what really a sleep here does is it puts an entry in the in the journal and says here is the time until which I want to sleep And then after a crash it would also not always push this into the future. But even better, if the sleep time is too far away, you can actually just throw an artificial failure and say, there's no point actually staying around because I'm going to sleep for very long, I'm just going to throw a failure. And because I've recorded that timestamp until which I want to sleep, a smart queue understands I don't actually have to resend the event and re-trigger the execution until that particular point in time has been reached. So what you effectively get is something like this here. You're doing the first attempt, you're doing a sleep, and then you basically just pause. You stop the execution. And depending on the programming language in which the SDK works, this is implemented in different ways, like in Java, in Kotlin, for example, it works very nice. You basically just stop and cancel the coroutine context. In Java, it does It does it actually with a very special exception that it throws to basically abort the execution. And then it waits for the time to be up. And after that, it basically restarts it. It replays. It understands, OK, the sleep has been there. Let me try and make the second attempt. So that's actually nice, because it means we're not actually keeping these ones around. We can actually use that resource that Yeah to process the next event and Yeah, as expected this also results in In a much faster processing time here compared to the the previous one right like even though we have those weight waiting effects it actually goes goes through reasonably fast the speed with which it With which it basically stops the execution and goes into and goes into this suspension mode where it says I'm going to go away and I'm waiting to be recovered when a certain timestamp is up. That depends very much on the configuration. How long do you want to keep this running or not? Okay, let's try and do one more code sample. And that is we're trying to do a request response kind of thing with queues. Let's say you have a service that processes an event, but it needs to gather additional data. For the sake of this example, we're trying to not do that by making, let's say, an HTTP call to a system to gather event, but we're sending an event to another service to say, you know, do something with that event, enrich it, and then send it back to us, and then we continue. So it's kind of classical, like sequential request-response style RPC. It turns out if you actually implement this manually, so this is basically a way to say we're doing reliable RPC in a decoupled way. Like senders and receivers don't have to be online at the same time. As soon as the sender has actually kicked off the request, it's going to happen and it's going to come back at some point in time. And I have completely decoupled availability and so on. And if we actually implement this manually, it's not a terribly easy thing because we're kind of have to start implementing a state machine now where our code says, OK, I'm starting, I'm receiving the event. Am I already in the awaiting response state? No. OK, then let me persist the state, send the event, and that's it. And then when I come back, I'm awaiting the event. OK, I get an event back for that specific key or correlation ID. I'm in the awaiting response state now. OK, let me reload the previous one, pick this up, process it, and send the result. This is a way to sort of implement it. to get it completely fully asynchronous resilient. But it's not really a convenient way to implement it. What's actually a much more convenient way to implement it is if you could write it just like this. Basically saying... We're doing something like creating an RPC client. See that we're creating this from the context here. This is, again, the sort of durable execution context that allows us to access the journal. And we're basically just doing an RPC call and await. And the interesting thing is this actually works as well, just like that. And why does that work? And why is that actually giving us the same sort of resilience and robustness as if we had created a manual state machine and persisted the intermediate state in the database? It's because this is again basically all we need is just our journal again. You know when we're creating this RPC call, we really just need to place actually in our journal the event that represents this RPC call. Just going back a little bit to this notion here. Assume this is here like an RPC operation. We really just need to place the RPC event in the journal, which again makes it to that point here. And this is the journal being in the event is the equivalent of the state machine that we created basically manually before. The one that says, okay, I've actually made a call, so next time an invocation comes, I know I've already made that call. The interesting thing is at the same time, because it goes into the same queue, it might the same journal event also just... functions as the RPC event. So you have basically done two things at the same time here. By creating this RPC call on the client, you've effectively created both an RPC call and recorded the fact that you've made the RPC call. So you won't actually do it the next time you've basically persisted your state machine. So we can actually quickly run this, but it's just going to run like the other examples. You're going to see some events going through. I think the more interesting bit is sort of the The observation here that the fact that you have the sterile execution journal in the background, it effectively is a state machine. It's an implicit state machine that runs in the background of your application and that turns your code into every time you complete a step, you're implicitly going to a new state of the state machine. So you get the ability to sort of build this event-driven applications, but you're not really working with events and state machines. You can have the system automatically maintain the state machine for you in the background. The journal is the state machine. All right, I have a few minutes left, not too many, so let me go through those parts real quick. I want to show you just a little bit of how far you can actually take this. So this is maybe just one thing is worth mentioning. The core... of a durable execution system is being able to run this durable execution state machine very fast. I think this is where the different implementations take their unique approaches, like Tempoil would do that slightly different than Reset and so on. In our case, we really tried to build a system optimized for making this as lightweight and as cheap as possible by having a single binary that integrates the actual durability layer. So it's not sitting on top of anything like Kafka. It has actually its own very lightweight log. and state machine and state orchestration and everything together. Let me skip over those parts except maybe here. The way you would typically use this, again, this is not very restate specific, but like most durable execution systems would work. You have your services that you implement on your typical infrastructure. You connect them to your durable execution engine. Typically, the durable execution system becomes something like a proxy or a gateway or a dispatcher or something like this. for the actual functions that you can call from the outside. They use their internal IQs and state machine to persist the fact that these functions should be executed like workflows. And it will also hold on to some sort of connection to actually stream back the journal entries. Just to quickly mention how far you can go with this. Here's a small example that we implemented where this is kind of a like, you know, door dash vault. Foodora style application where you can order food and it coordinates drivers that deliver this to you. And this has a bunch of microservices if you wish. There's the workflow that does the order processing that has to talk to the payment provider, restaurant. There's a delivery service, a matchmaking pool between drivers and orders. There's a digital twin for the driver that... that basically tracks where the driver is and manages the orders that are assigned to the driver. If we would like to implement something like this in... If we want to implement something like this, we probably do want to worry a little bit about the resilience and reliability, because an order that has been started shouldn't get lost, right? Like a driver shouldn't get two orders assigned, or it shouldn't be that our order gets assigned to the driver and then there's a crash and it gets orphaned or so. So those are actually... Those are actually great cases for such durable execution engines. These type of applications where you care about the fact that something happens. Let me quickly stop this bit here so that I can actually bring up the other demo. Okay, nothing running here. Good. I'm going to run most of the microservices from the IDE and then all the auxiliary bits from Docker Compose. The auxiliary bits being the reset durable execution engine itself, but also Kafka. Actually, the picture is slightly wrong. I changed it last night. The orders basically go directly to the workflow service, but we're using Kafka topic to let the driver apps send periodic pings. I'm here that are consumed by the app to say, okay, how far are you still away from your order? How much do you still have to drive? Have you reached the point to pick the order up? Okay. This should be up and running now. So if we open the tab here, we should see that we're able to get one of those orders delivered, the driver's on the move. The interesting thing here is if we look at how this order workflow, for example, is implemented, it's an extremely easy bit of code. It's just a few steps here that the top is you can almost ignore this It's just like an RPC client again through sort of the durable execution context that Goes to some form of like status service where we can just put the status or that the web front end can can query it I think the more interesting things are at those bits here Payment processing the first thing we want to do if we you know If we do this, we want to make sure we talk to Stripe to charge the credit card. The first thing we need when we talk to Stripe is actually a deterministic ID. So if the network call falls in the middle and gets retried, nothing is charged twice. So we can actually use the durable execution context here to generate a token which actually goes to the journal as well. So the same payment item potency ID is actually used on every retry. Then you can just hook the payment process itself in one of those operations so it gets journaled and once it's complete, you understand, okay, we've actually done this, you're not retrying this. The next thing is if you schedule an order for the future, you know, you're trying to order something for in two hours when your parents come, then you can just use one of those durable sleep operations here to say, you know, delay the order before you actually talk to the restaurant. I think the next bit is also very interesting because you very often have the characteristic where you're not just dealing with your own workflow itself, but it has some interaction with the outside world. So for example, the restaurant should notify us when the order is ready, right? So basically we can create something in the durable execution journal. And again, this is the syntax in reset, but it basically is the same thing in all of these systems. can create something like a future here. It doesn't have the return value. There's no actual value in there, so it's like a void serialization that we're passing in there. But it will have an ID, and we can tell the restaurant that ID here when we're making the call. Prepare the order ID and the ID of the future. And then the restaurant can basically make a webhook call with that ID. And this basically gets then injected into the same queue, gets put into the same sort of like sub journal, and it makes it sort of back into any like recovery invocation that we're replaying the journal. So we can then just say, you know, await on this future. And we actually have a durable form of a future here. It's not a future where if the process crashes, you know, like that future is basically gone, but we have like a durable version of a future here that After any crash and recovery on a different machine, or let's say the restaurant takes two hours because you just ordered like a seven course French meal or something like this and the container gets migrated in the meantime, when it comes back after a failure, the value of that future will absolutely be there because it comes through the durable execution journal. It doesn't come through an actual sort of like transient network request. So we've kind of hooked this up in a, we've kind of hooked all those things up in a reliable manner. The final thing I want to say, we can actually, probably running out of time, otherwise we could play around with a few simulated crashes here. This code that we've written here, you can almost not make that system fail. You can crash it at any point in time, and it will perfectly pick up where it was before. It will not lose any callbacks. It will not lose any orders or any intermediate status. And the code that we've written is foremost trivial in that sense. We really just wrote down the sequence of steps. And we're letting the durable execution system basically wire everything up for us and the reason why that works is This fine-grained durability like every operation that we do here is a durable operation every step is persisted every future is basically persistent and so on and The the trick about durable execution if I can say this is like a final summaries. I think in two things like number one Making this fine-grained persistence so cheap that it's actually feasible because There is a bit of work going in the background, but you can actually, like Reset gets these operations like to single digit milliseconds or so. So you're adding a very moderate overhead for each of these durability operations. And the second part is the way of fusing the operations and the journal together in such a way that basically you have sort of this automatic... atomicity of operations. And what do I mean by that? That the creation of the future, for example, and the registration of the callback ID, the creation of the code and the registration of the callback ID in the system, they're always atomic steps. For making an RPC, the fact that we're sending the RPC and the fact that we're in the recording that we made this, they are atomic steps. There's not like two different things that we have to worry about in the, like after recovery that, you know. We might have made the call, but not recorded it or the other way around. This is basically the essence of what it gives you. It gives you this lightweight way to do very fine-grained persistence in an SDK, in a protocol that makes these operations consistent by itself. That's basically it. Let me stop here. I don't know if we have much time for questions, but I'll be here after the talk.

Subscribe to our channel: https://youtube.pl/c/DevoxxPoland?sub_confirmation=1

When building resilient distributed applications, event-driven applications and logs like Kafka have become one of the most popular choices. But in the recent years, a new kid has appeared on the block of event processing tools: durable execution engines (e.g. Restate, Temporal). 
In this talk, we will take a look at event-driven apps and durable execution engines differ and when to use which one. 
The key topics are:
 - How do both models provide reliability of application logic? What does the developer need to do and take care of?
 - How do they manage asynchronous tasks, decoupling of synchronous parts from and long-running work
 - How do applications manage state?
 - How are they deployed, how are they operated (clusters, deployment, scaling)
 - How can applications be observed and debugged?
 - What typical use cases and sweet spots do the different approaches have?
Join this talk to learn how each technology tackles these aspects, and what their main differences are.

Recorded at Devoxx Poland 2024

Twitter: https://twitter.com/DevoxxPL
Instagram: https://www.instagram.com/DevoxxPL

Join us also here:
Devflix: https://devflix.pl

#Devoxx #DevoxxPoland #IT #Development #SoftwareDevelopment