Failure Is Not an Option: Durable Execution + Dapr = 🚀 - Marc Duiker, Diagrid

All right, we are good to go. Welcome everyone to my session, failure is not an option. Drupal execution plus dapper is rocket emoji. I think we can translate it as amazing. Yeah, thank you for spending this half an hour after lunch with me. I think it's my job to keep you awake. Hopefully I can manage that. Let's get going. I think we are pretty much familiar with failure in IT, right? Failure is inevitable. I think we already, everyone experiences it probably firsthand. Maybe we caused even some failures, I don't know, but definitely as end users, we probably are familiar with a lot of IT failures. Usually it's always DNS that's causing it. I don't know why. And maybe some of you are as old as I am and are familiar with this Windows XP message, information message that says, task failed successfully. which is of course a very weird thing to show to your users. But that's actually what this talk is about. Failure is inevitable. Things will fail. So why not make sure things will fail successfully? So maybe we can even automatically recover or maybe we can make the impact on the user as small as possible. So I'm Mark Duiker. I'm a developer advocate at Diagrid, a company founded by the co-creators of Dapper Open Source. I'm one of the Dapper community managers, big fan of pixel art, as you can see. Also a I'm a big fan of VS Code. I'm running my entire presentation in VS Code. I will share a QR code with the GitHub link later, so you can definitely see all of the slides and all the demo code yourself later. I did a bit of research about IT failures and I came across this very interesting blog post. It's already quite old. It's from 2015. It's on the IEEE Spectrum blog post, but it describes lessons learned from a decade of IT failures. It's quite a long blog post. took one screenshot out of it. But there's this legend and the bigger circle is like 10 billion of monetary costs of IT failures. And here in this diagram you see across like 2005 and 2015 across the whole globe, like very big circles. So you can imagine the monetary cost of IT failures is running in the hundreds of billions. And this was like 10 years ago, but I'm pretty sure that maybe the cost is even higher these days. So the link is all in here. You'll get the QR code later. So definitely check out this link yourself. It's very nice. And it's also an interactive blog post. So that's great. Yeah, one of the reasons maybe IT failures are a bit more prominent is because also our applications are becoming more complex. So this is even not a very complex thing. We are making more and more distributed applications and services need to communicate with each other, either synchronously or asynchronously. There's lots of message brokers involved, lots of different state stores involved. There's a lot of moving parts. There's a lot of things that can go wrong and that will go wrong. That's nothing new. Who heard about the fallacies of distributed computing? Okay, so some some hands so this this list of false assumptions is compiled in 1994 so quite some decades ago So it's pretty much well known that distributed computing is hard and that we don't understand everything when we start working with it I'm not going into detail all of this because this is not really a theoretical session about this I can definitely recommend you with some of these links that I've included here and one of them is like a playlist where Uri DeHaan from Particular Software describes all of these fallacies in very short videos So definitely have a look at that. Now about five years ago, some smart people got together and created this open source project, Dapper, the distributed application runtime. And over the years, it has been used by really hundreds of companies across all kinds of verticals to really help developers really run and build their distributed applications. So it really makes it much, much easier to build, but also run it at scale. Last November, it became a graduated project within CNCF. So, of course, the Dapper project is really, really proud to achieve that. And on the CNCF website, also on the Dapper website, there's lots of case studies mentioned on how companies are using Dapper and how they use it successfully. So, yeah, if you have any doubts about Dapper, definitely have a read in these use cases. So, anyone familiar here with Dapper? A couple of people. Who's using it like in production right now? Okay, definitely stop by me later. So Dapper runs in a sidecar next to your application and you can use any language since it runs in a separate process. And Dapper offers a lot of APIs that you can use as a developer to quickly build your microservices. Now I'm not going to talk about all of these APIs because this is really a session about durable execution and workflow. Definitely, I recommend you if you're new to Dapper, I can recommend you to try out Dapper University. I'm working on some educational content about Dapper now. I will also share a QR code at the end of the session where you can access this. It basically runs like a sandbox environment in a browser, so you don't need to install anything to try out Dapper. So getting back to the problem that we're solving, so we know that systems fail and we need to recover from those failures, ideally automatically, but also limit the impact of these failures. So and one of these solutions is durable execution. So durable execution is means like running code in a stateful way, which sounds a bit weird. Well, we've actually doing that for quite a while. So in case the process that runs the code, if that process fails, another process can come up and it can read back the state because that state is persisted on disk somewhere and it can then run the code to completion. Now, like I mentioned, maybe this job execution term is maybe a bit new, but maybe you're familiar with workflow engines and workflow engines have been implementing this job execution stuff for many years. So who've been using workflow engines in general to automate their business processes? Yeah, I see quite some hands, right? If you're new to workflow engines and workflows in general, a workflow consists of many tasks or activities, and all of these activities are small units of work, maybe calling out to another API, saving some state to a database or retrieving some state, publishing a message, and so on. Your workflow typically consists of all kinds of activities that make other calls and some business logic. The way that workflow engines and durable execution work is because all of these state changers, they are stored to disk. to a state store. So I've created an animation just to give you an indication how this works in general, because there's actually a lot going on. So as soon as you start a workflow, the input of that workflow is actually stored in the state store. So there's a workflow ID and input that's stored. Then the first activity gets scheduled. Well, that first activity is then being executed and the input and output is also stored in that. database so then we don't continue immediately to the second activity but the workflow will start from top again so it will read the stage from activity one it doesn't execute it again because it's known that it is already is there then it goes to the second activity and again everything is persisted to disk and again we replay from the top before we go into activity three so as you can see there's a lot of io going on between the workflow engine and and the state store and yeah this is necessary because yeah a workflow uh can can stop can can break any time but all of your uh had your status workflow is persisted to disk so had this in general how these these workflow engines and turbo execution works now there are different ways how you can author workflows right so when typically and had the system started with visual designer so you can more click and drag all of these um these these workflows they're also like um step functions which is more like json based and but they're also like real workflow as code solutions I'm a big fan of these workflows code solutions because it's part of your source control. When you submit the PR, it gets reviewed by someone else. You can unit test your workflows, which is ideal because workflows should contain business logic, which is quite valuable if that's correct. I'm a big fan. I have some examples of workflows code solutions like Temporal, maybe people are familiar with Azure Dribble functions if you're doing serverless. Now, and Depa workflow is also like a workflow as code solution. So the Dapper workflow API has been stable in since the last Dapper release. So that happened in 1.15 in February and it's been like under development for like over two years. And if you want to know more about these drill execution engines, I also include the list to an awesome GitHub list that contains like dozens of these different workflow engines because there are much more out there. So when you start using these workflows code solutions, you can you can use different workflow patterns with these and with these things so the most basic pattern that you can use is called task chaining and which is basically you one run certain activities in a specific order and you need the order because maybe the output of activity a is used as an input for activity B and so on and so the order is important Another pattern is the fan outfit in pattern and here there's no dependency at all between activities You can run them in parallel or in large batches The nice thing about this though is that your workflow can actually wait until all of these activities are completed and then you can aggregate over the results of all these activities. Another pattern is the monitor pattern. This is great when you want to do a reoccurring task. Maybe you want to run a nightly clean up job in the cloud and remove some cloud resources. The first thing you can do is implement a call to your cloud to clean up some activities. Then you can instruct your workflow to create a timer, wait 24 hours. The workflow will unload, it will not do anything. Then after these 24 hours, it will become active again. Then you instruct it to restart as a new fresh instance. the same as the replay behavior I showed earlier because when you do replay it will remember all of its history executions but if you say continuous new it's a new fresh instance final pattern I want to mention is the external system interaction so this is great and we want to implement a approval workflow for instance and so here you can instruct your workflow to wait until an external event comes in and then you can use the payload of that external event to decide how you want to continue in your workflow And normally you wouldn't use just one of these patterns in your workflows. You would actually combine several of these patterns. So this is an example where we use a bit of task chaining. We are waiting for an event. We also do a fan in, fan out one. And the nice thing, of course, if you use Dapper in general, you can use and Dapper workflow, but also all of the other Dapper APIs in those activities. Then a bit about the Dapper workflow engine. So the Dapper workflow engine is part of the Dapper sidecar. So as soon as your application starts up and a Dapper sidecar starts up, there's a GPC stream that's established between your application and the Dapper sidecar because a lot of communication needs to be happening for scheduling your workflow. And your application itself will still run the workflow, but the Dapper sidecar is responsible for scheduling it. And in case it all goes down, the Dapper sidecar will actually start up your workflow again and will start executing all of these things. And of course, the workflow engine is responsible for persisting all of that workflow state to the state store and getting that back into your process. All right, almost time for our demo. So I have a demo with two applications, a workflow application that talks to an inventory and have a shipping application. And we'll do some kind of a simulation of an order process where we sort of validate an order and interact with the shipping service. Now this is what the workflow is doing. So the first activity of that workflow is an update inventory where we actually check do we have enough inventory and if you do we actually mutate that inventory. If you don't we immediately exit. If we do have enough inventory, then we have another activity that gets back the shipping providers for this product that we're ordering. This gets back an array of different shipping providers. And then for each of these shipping providers, we will retrieve by making an API call to another service that gives us the cost for this shipment. So this gives back like several cost indications. And then in our workflow, we have a bit of code that actually chooses what is the cheapest shipper for this product. And then for the cheapest shipper, we will register this shipment. If it's all okay, then we end. If this is not okay, that means we can actually do a compensation action. So we can actually undo our update inventory again. And a compensation action is not something that is... built in into a DEPA workflow. It just means you write an activity to undo something that happened in an earlier activity. So we are actually undoing this update inventory when we call this activity. OK, let's have a look at the code. In this code, in this case, I'm showing C Sharp. But you can write your DEPA workflows also in Java, JavaScript, Python, or in Go. So in this case, we just have a class we inherit from a workflow class that's given by the Dapper SDK. And we are running this method called runAsync. So the Dapper SDK gives us a workflow context and we'll be using this context to call out and to schedule our activities. We give it an order as an input. So that's our own object that we provide. And the output of this workflow is an order validation result. Again, that's just a custom object that we define ourselves. So we are using this context to call to an activity. So we instruct the Dapper runtime, okay, now run this activity. And as soon as the Dapper workflow engine will actually schedule this activity, this code will stop executing. It will be unloaded from memory. And once this activity code has been completed, then this code will actually start running again. So that's the replay nature I showed in the animation earlier. So we call an activity and you always refer to activities by name. So in this case, we're calling the update inventory activity. We pass it a payload and we get back an inventory result. And then we have just some if statements, which is our business logic. So if there's a sufficient stock in our inventory, then we continue. The next activity that's called is getting the shipping providers and here you see I'm adding in an additional argument that's called a workflow task options So you can actually provide a retry policy when you call these activities I didn't do it in the first example here But you can actually instruct the workflow in case this activity fails retry it a couple of times And in this case, it's just a simple constant retry policy, but you can also instruct it to like exponential back off. So that's totally up to you. So this gives us back an array of shipping providers. So what we're going to do next is we want for each of the shipping providers, we want to ask them, okay, what's the cost of your shipment? So that's what's happening next. So we are iterating over the shipping providers in this for each loop. And again, we are calling an activity get shipping cost. But notice that we are not awaiting this call here. So that means the DEPRA workflow engine is not actually scheduling these calls yet. We are purely adding these tasks here to a list. So this is a list of tasks of shipping cost result. And at this point, we are actually scheduling all of these tasks and also instructing the workflow here to wait until they have all been completed. So this is the fan out fan in pattern here. So once all of these activities have been done, we just ask, hey, give me the cheapest shipping service. So I just just gives me the minimum and the minimum cost here. So at this moment, I have a shipping service identified, which is the lowest cost. And then the next part is registering the shipments for this shipping service. Now, I've wrapped this in a try catch class to just to indicate that you can use some other control logic inside a workflow. So. I'm calling a register shipment activity here. But what happens if an exception is thrown in this activity? Well, we can then catch the exception. Exceptions are wrapped in a workflow task failed abstraction. In this case, you can, of course, always inspect what the inner exception is of this one because this is a really workflow-specific exception. But this is how, for instance, you can determine if you need to execute a compensation action. called this activity fails. Then we are in the sketch block and then we do the undo update inventory activity. This is our compensation action. Just to show you an example of an activity. There are multiple activities, but I'm just going to show you one because they all have the same structure. Again, an activity is just inheriting from a workflow activity that's based on the Dapper SDK. In this case, the inventory is actually checking against a state store using the Dapper API. That's why we're using the Dapper client here. We have a run async method again, which is being called and we're using the Dapper client to retrieve some state. So we get back a product inventory and we just take a faith. Do we have enough of this product in stock? If not, we of course give back a result that we don't have enough in stock. But if we do, then we make a mutation to this state and we save this state again using the Dapper client. All right. Now, this is the startup part of the program. So what's important for Dapper workflow and what might be different with other workflow as code systems, you need to register your workflows and activities with Dapper. Because otherwise, if you don't do this, Dapper has no clue that there are workflows or activities present in your code. So in this case, we need to use the add Dapper workflow method on the service collection where we need to register our workflow and our activities. Now, when I run a demo, I will run the execute this endpoint. There's a validate order endpoint. And what happens here, we are retrieving a Dapper workflow client and we are executing the schedule new workflow async method. So here we are instructing the workflow engine, please go ahead and schedule this new workflow. Again, you do it by name. So we pass in this work, validate order workflow. We provide an instance ID because each workflow needs an... Instance ID if you don't provide one it will just generate a good for you But in this case I'm just using the order ID as my instance ID for this workflow and we provided the order as input for the workflow So notice that that we don't get back to the result of this Validate order and and footage for this workflow, but because everything is asynchronous So what we actually get back here is the instance ID and the intercede in this case is exactly the same as your order ID. So I am I could have just returned the order ID here, but if you don't provide this order ID, then you need to get back this instance ID. Otherwise, you don't have anything to correlate to what correlates to my order. So we don't get back the result because we are triggering a potentially very long running process. In this case, it's very quick, but this could also take hours or days or months. So that's why you don't get back the result of your workflow because everything is asynchronous. I have some other endpoints to update the inventory, but I think it's almost time to run the sample now. As I mentioned, durable execution, workflow engines, state needs to be persisted somewhere. You define what kind of state you use via component files, if you're familiar with Dapper. You define them in this YAML structure. In this case, I am using a state store that's based on Redis, because if you use a Dapper CLI, you always get a local Redis container running. um so redis is where where our state is stored and you're not directly interacting with the state we're using the the dapper apis to interact with this you're not interacting with the state directly yourself and dapper workflow is built on top of dapper actors again you're not using the actor api yourself when you are working with workflows it's just something that is useful for you to know and that's why you need to make sure that the actor state store flag needs is set to true so this is like a prerequisite if you want to use workflow Right just one more YAML file I promise and then we're done and so I'm using Dapper run to start up my services locally and in this case I have configured which applications to run in this YAML file I'm starting up a workflow application and I'm starting a shipping application All right, so what I'm gonna do now is I'm gonna use Dapper run I'm going to use this command, but it's all scripted. So you won't see me typing. I just click a button and this will run. So this will run both applications for me. It will run the Dapper sidecar for me as well. So once this is running, I will use a REST client within VS Code to make a call to this validate order endpoint that will start my workflow. Then I will stop all processing to indicate something severe has happened and our applications are crashing and burning and not alive anymore. And then I will restart the workflow with this Dapper run command. And then we'll see what happens. because we were actually aiming for automatic failure recovery, because that's what your execution is offering us. All right, so I'm running the app now. So now our workflow application is starting up. Now our shipping application is starting up. Also, the Debra sidecarters are starting up. So everything is running now. Now let me make sure that we first have some inventory. So this is working. So let me check if this works. Yeah, okay. So I have like five rubber ducks in my inventory. So that's all working. Okay, so I'm now calling the validate order endpoint here and this is my payloads that I'm providing I want to order two rubber duckies. Okay, so I am sending the requests and Then I'll wait very briefly and then I will stop everything Yeah, I will stop right now. All right, so the last thing in our logs that we've seen I mean there's a lot of debug logging here But the last thing that was happening is our workflow application was Making a service call to our shipping application and that's the last thing about what's happening here So this is a log from our shipping service and that is going to retrieve the shipping cost for this order But before it could return this information to our workflow. I have stopped all applications right to to indicate something something bad happens So everything is down workflows down shipping app is down. So let me let me restart everything again So now I'm running Dapper run again, which just starts up everything and let's see what happens Okay, and again we see that the shipping services are contacted again all three of them now and let's wait a bit more and soon and now the chipping is the cheapest one is selected that's Service B in this case and now we see a log statement called orchestration status completed, right? So this is actually the Dapper workflow engine completing our work after a failure. We only need to start up the process again, which if you're running in Kubernetes, Kubernetes will make sure you'll get a new part and it will start up everything again. But I didn't need to make a call to this valid order endpoint again, right? Because Dapper realized, okay, I have a workflow that is not completed yet. So let's run it again. I can verify it stayed by making a get request and with this instance ID to get back the state of this workflow. This is the workflow. This was the exact instance ID that we have run and this is the final state. This was the input and this was the output. All right. This is how Dapper uses the turbo execution concept. Okay. Let me stop executing this. All right. I will... skip the next demo due to time but i do have some other important parts to mention all right and so workflow as code is great or roughness in general are great but there are still some challenges when you're working with workflows so there are four points i want to discuss these are all described in great detail in a blog post by chris gilliam i definitely recommend you to read this blog post if you're really really seriously considering to use it to use workflows One of them is that your workflows should contain deterministic code. Let me zoom in a bit more. So why does your code need to be deterministic? Well, your workflow is being replayed several times, right? So if during this replay, your behavior changes within the workflow, this can cause a mismatch between the state that is stored and what's happening during runtime. right because after each activity call all of the inputs and outputs of this activity call are stored into into this into the state store and uh when during a replay when some of these arguments changed there there's a mismatch between between the states and then you yeah you have like a very weird mismatch between stating workflow so um what you should not do for instance is this you should not use like um good new good which creates a new random good inside your workflow or you should not generate a new timestamp in your workflow and have for these like common use cases there are actually some some some helper methods on the workflow context called new good and also context current and you can see daytime so and there are some helper methods to make that a bit easier if you have other codes that is non-deterministic just wrap that code inside an activity call because your activity codes can be non-deterministic, that's totally fine, but make sure your workflow is deterministic. Now, your activities, the DEPA workflow guarantees at least once execution of your activities, meaning your activities can be executed more than once. So what happens when you write your activity code where there's not a guarantee that there's different behavior in your activities? So for instance, what if you do an insert in a SQL store and when you provide a primary key? So what happens if you do an exact same insert with that primary key for a second time? Well, that probably would not work, right? So you're probably then better off to do like an upsert or you first do a read before you do a write. So yeah, you need to check what kind of code you're using. Other APIs, maybe the APIs that you're using are idempotent, maybe not. So you have to make sure that the code that you're using is idempotent. Another tricky thing with workflows is versioning. Basically, whenever you make a significant change in the workflow, maybe you change the order of the workflow or you change some input arguments to some other arguments, that is a breaking change in the workflow. And that's, again, it has to do with the difference in between states and what you're using at runtime. So for instance, I have this first version of a workflow and that has the order item, which I'm getting as an input for my state. I'm using that as an input for both my activities. But then I change my mind and the second iteration of this workflow is I'm using the order item as an input for my first activity, but I'm using the result as my first activity as an input for my second activity. Now, when I deploy this, and there are still some workflows in flight, meaning some uncompleted workflows, they have the state that belongs to this version, meaning they have the state that contains this order item as an input. But now my new code expects the result A, which is actually a completely different type. So that's a breaking change. So I think there are several workarounds to this. Probably the easiest is to just version your workflows, right? So, and here we see... the workflow name is really has a number version one version two yeah so that's so never change the workflow just always add a new one it does mean that your client that's calling the workflow that also needs to be updated because your new version should now start calling this one instead of this one but that's the easiest thing Okay, final one. Just make sure that when you are passing arguments between activities, make sure they are as small as possible. Because all of these arguments, they get stored to this taste store. So in this case... We pass in an ID, which is small, which is great. We get back a large document from this activity and we're passing this large document as an input for another activity right here. So that means that this document is saved twice in the State Store. First, it's saved as an output argument for this activity and it's saved as an input argument for this activity. So it's very inefficient. So in that case, don't make your activities very small and granular and give a bit more responsibility to your activity. So just make sure in this case, do both a get and an update and a save in one activity instead of like making very micro activities. All right. My time is up. I hope. I'm showing you a bit about about the workflow and got to our codes for you to share. So the left one is this GitHub repo where all of the slides are and all of the source code is the right one is a link to Debra University. You can use that if you're completely new to Debra, but in a couple of weeks, I'll release a new lesson that's completely about Debra workflow. If you have questions, just come see me afterwards. Thank you very much.

Don't miss out! Join us at our next Flagship Conference: KubeCon + CloudNativeCon events in Hong Kong, China (June 10-11); Tokyo, Japan (June 16-17); Hyderabad, India (August 6-7); Atlanta, US (November 10-13). Connect with our current graduated, incubating, and sandbox projects as the community gathers to further the education and advancement of cloud native computing. Learn more at https://kubecon.io

Failure Is Not an Option: Durable Execution + Dapr = 🚀 - Marc Duiker, Diagrid

Applications break all the time, there could be a network issue, a cloud provider outage, or just a glitch in the matrix. But as a developer, you really need your applications to be resilient without the need to recover databases and restart services manually.

In this session, I'll demonstrate how Dapr Workflow provides durable execution, which enables you to write reliable workflows as code. In addition, I'll show how resiliency policies in Dapr improve reliable communication across services and resources when developing distributed applications.

I'll go into specific workflow features, such as scheduling, sequential and parallel execution, and waiting for external events. I'll show many code samples (in C#) for each of these features and will run the applications using the Dapr CLI to demonstrate their resiliency.

By the end of the session, you will have a good grasp of how durable execution with Dapr workflow and resiliency policies can help you build resilient applications.