Failure is not an option: Durable execution + Dapr

Hi everyone, welcome to this session. Failure is not an option. Dual execution plus dapper is amazing. If you're working in IT, then you probably know that failure is inevitable. You probably see it every day. We see like new bugs appearing every day that we need to fix. There's like outages at cloud providers. It always seems to be like DNS. Failure is everywhere. So we need to take that into account. And maybe you're as old enough as I am and that you've seen this information message from Windows XP a long, long time ago. And it says, task failed successfully, which is, of course, a very weird thing to show to your end users. But the message is actually what it's about in this session. Things are failing. We better ensure that things are failing successfully and limit the impact so end users might not even notice that there were failures. I'm Mark Duiker. I'm a developer advocate at Diagrid, a company founded by the co-creators of Dapper Open Source, and we build solutions in the Dapper ecosystem. I'm one of the Dapper community managers. We do live streams every month, and we also manage the Dapper Discord. So if you're not there yet, please join the Dapper Discord. I'm also one of the Microsoft MVPs and I do love knowledge sharing with Twist. I also like pixel art a lot. So that's why you see this up here as well. If you want to know more about my creative things, have a look at markduiger.b. When I did some research about... IT failures, I came across this very old blog post on the IEEE Spectrum blog. So it's a very respectable blog site and it's called Lessons from a Decade of IT Failures. And this is a really nice blog post. I highly recommend everyone to look into that. We will share the links later as well. But it looks into all kinds of IT failures between 2005 and 2015. And it's also like a very interactive report as well, as you can. you can drill down into some some of the of the issues in that report i just took a screenshot of the first one um where you can explore what the monetary cost um has been between 2005 and 2015 and well the legend here um with all these circles indicate how big the cost is and so the biggest one is worth 10 billion in it cost and if you scroll down here then we see it by region and also here on the x-axis at the time so in this 10 years there were like a lot of very big i.t failures which cost like over the hundreds probably even thousands of billions in in it cost uh and this this may not be a surprise for you But the scale and the size of this is really impressive. And I'm not saying by following the tips that I will share in this session, this will be completely gone. Definitely not. There will always be IT failures. There's many reasons of IT failures. But I think a lot of these failures can either be prevented or the impact can be limited. All right. So a lot of organizations are building. distributed applications. So applications that involves a lot of different services, different state stores, maybe even different message brokers. And these are quite complex things to build. And it's not that organizations want to build distributed applications, but typically they are forced to do it because they have like dozens or maybe even hundreds of development teams. Maybe they're using different technology stacks. So it's inevitable that you end up with distributed applications. And we've known for quite a while that it's actually hard to build these distributed applications. So there's this fallacies of distributed computing, which is quite well known, but that's already from 1994. And that is a set of false assumptions programmers new to distributed applications often make. I will not list all of them as it's not a very theoretical session. I definitely recommend you to look into the links, especially the last one, because that's like a YouTube playlist where... Udi Dahan from particular software gives an explanation about all these fallacies. So I think it's very, very interesting to look into these a bit more. It's also very good to understand there are different types of failures. And one of the axes that you can look to categorize failures is are they temporary or transient or are they permanent, right? So transient failures are temporary and they're usually resolved themselves without any human intervention. It's good to realize that when that happens, it means that someone else before you has thought about this specific edge case and made it so it's not an issue for you anymore. So maybe someone fixed something in hardware or in firmware or in software that things will keep on working eventually. And it's also the area that we'll look into a bit further. The other type of failures, those are permanent failures. So those are the opposite of transient failures and they do require. Simon Ruetz- Human intervention to be resolved, so you could think of maybe someone introduced a bug in the source codes well that's not going to be resolved automatically someone needs to go into that source code update it and you need to rebuild and redeploy it so that requires human intervention now looking at and distribute applications, there is a. open source project called dapper that stands for the dispute application runtime it's been out there like almost six years now it's a graduated product within cncf and it's been used by a lot of enterprises to build distributed applications and it's really proven itself over over the years so large organizations use this and are very successful and one of the benefits actually is that developers are also much more productive when they use something like dapper versus not using dapper So how does Dapper actually help you building these distributed applications? Well, Dapper actually decouples your application layer from the underlying infrastructure layer here at the bottom. So you can write your application in whatever language that you want. Dapper sits next to your application, so it uses a sidecar model. And that really helps you to quickly build applications that, for instance, need like pop-up messaging or server certification or do some state management. all of these functionalities are built in into the sidecar so you don't have to write that into your application you just need to make you to invoke these functionalities on the sidecar another benefit is that it also provides built-in observability security and resiliency so again you don't have to include all these features yourself in your own application because the sidecar will take care of that so hey you shift some responsibility towards your sidecar so you can focus on your business logic in your own application now as you can see the depper sidecar offers lots of different apis and and features in this session we're only looking at at the workflow one but it's one of many different features if you're unfamiliar with deborah i can definitely recommend you to have a look at depper university um we offer this for free and the only thing you need is like a name and an email and then you get a cloud-based environment like a sandbox environment where you can run dApp applications and test out how it works. So please try that. So going back to the problem that we're solving, so we know that systems are failing, we need to recover from failure, ideally automatically, so these transient failures, and we need to limit the impact of failure, right? Because it's inevitable, so if we fail, let's make sure that we limit the impact. So one of the things that can help us are JupyExecution systems or workflow systems. Now, what is JupyExecution? JupyExecution guarantees that code is run to completion even if the process that runs the code terminates. So JupyExecution ensures that another process is started and that the code is resumed until it completes successfully. And this is realized by persisting the state of execution to a state store. Now, if that sounds familiar, maybe you've used workflow systems in the past because workflow systems have been around literally for decades and they are used to automate business processes and they implement this durable execution concept. Now, workflow systems consist of three different things. So usually you have an authoring environment to create the workflow. Then you have like an engine which actually schedules and execute the workflow. And there is a stage store to persist that workflow state to make it durable. Let's have a look how most of these systems work. So here we see a representation of the workflow and there will be a lot of interaction between the workflow engine and the stage store. So as soon as we start a workflow, the workflow ID and input will be stored. Then the first activity is scheduled. So the input for that will be stored. This will be executed, but when it's completed, also the output will be stored. Then we won't immediately go to the second activity, but the workflow will replay from the start. It will get the information for the first activity from the state store. Then it will schedule the next activity. And again, it will store the input. It will store the output. Then again, it will replay from the top. It will hydrate the state back from the state store. And then it will do activity three. And finally, once activity three is done and that. state is also in the state store it will replay again for the final time and it will get all of the results back from the state store and then our workflow has actually come to completion and then also the workflow state and output of the completion will be stored in the state store so as you can see there's a lot of io happening between the the application the workflow engine and the state store so this is important to to realize especially when you start debugging workflow applications so one of the workflow engines is dapper workflow part of the Dapper project and it's using the Drupal execution concepts by persisting workflow state in a Dapper compatible state store if you're already using Dapper then it's very nice to use Dapper workflow because Dapper workflow works seamlessly with all of the other Dapper APIs so Dapper workflow uses a workforce code format so you write your workflows in the language that you're comfortable with so it's like C sharp or Java or Python JavaScript or Go And you can apply lots of different workflow patterns. So let's look into that shortly. Here's like a typical but small workflow example that shows you what kind of steps you can use in a workflow. So you typically start by receiving a message, either from a message broker or maybe it's just an HTTP call. Then you apply some business logic inside your... workflow in this case you call an activity and that uses a dapper api to retrieve a secret and then it makes a service application call to another service and then it will continue but there it will wait for an external event to come in once this event has come in the workflow will continue and then it will do activity b and c in parallel so it stores an event result using the dapper client so the dapper state store and it invokes another api And finally, in the last 50, we publish a message to another message broker, and then our workflow is complete. Now, I already mentioned that workflow authoring with Dapper workflow is via code. Personally, I like it a lot. I have a developer background. I like writing code. So I really like that you can author your workflows as code because that's very understandable to me. I also like it that it can be unit tested easily as you can really prove that your business logic works. And it's also part of source control. So you make like pull requests, people can review it so they can also verify that your workflow is actually sounds that the business logic is good in there. Downsides of... workflow as code approaches is that typically there's no visual representation of the workflow itself there are a lot of workflows code solutions right so in this session we're looking into depth workflow in a bit more detail but there's also temporal if you're familiar with azure there and azure functions there's azure durable functions if you know azure durable functions the api is like 99 the same as depo workflow so it's very easy to to go from dual functions to depo workflow because you will definitely understand the api and if you are interested in general about these triple executions or workflow engines on github there's a github repo called awesome double executions and that lists like dozens and dozens of these systems so you can do a bit of comparison if you want and of course all of these links will be will be put into the description so you can have a look at it Let's look into some workflow patterns because if you're working with workflows as code, there are some very typical workflow patterns that you actually need to use. And sometimes these things work a bit differently than you would program your application without using workflows. So the first one, and that's the most basic one, is called task chaining. So task chaining is where the order of execution of the underlying activities. is of importance and it is because maybe there is an output of activity a that's used as an input for activity b and the output of activity b is used as an input for activity c so that really requires this specific chain of these activities and by the way um your workflow typically consists of um calling these activities uh some if else statements um but that's it right so all of the uh code that you would do otherwise all of the non-deterministic code like calling out to do a database or calling out to a service all those things are wrapped inside an activity next part on pattern is the fan out fanning pattern and this is very useful if you want to execute multiple activities in parallel and which you can do when there's not no relationship between these activities right so there's no output that needs to go into another activity as an input and you can either call different activities um different activities but you can also choose to um for instance if you receive an array as an input you can also call the same activity but then for each input for the array so you can use this in different ways so it will do the fan out so it will schedule and execute all of these activities but then the fan in parts also very important so um dapper workflow will schedule all these activities so that it will run but the workflow will actually wait here until all of these activities have been completed and then you can still do some aggregations at the end with the output of all these activities the monitor pattern is very useful when you want to do some kind of a recurrence of an activity so for instance you want to run a nightly cleanup job to clean up some resources in the cloud well you can call your cloud api to do the clean job in this first activity so this makes a service invocation call or some other calls with third-party api in this activity then next you will use a timer in this workflow application to to wait we're going to wait 24 hours before we want to run again but now we are using the continue as new method and this instructs the workflow to start as a fresh instance and forgetting all of the historical executions so this is basically like writing a loop But in a workflow code, you should never do like a do while loop for instance, but just always use continuous new as a loop. Because when you use this method, you can always provide some new inputs that can be used inside the workflow when you do this loop. the final one is external system interaction that is very useful if you if you want to workflow to wait until an event comes in and so for instance maybe you have like a typical order approval workflow so maybe you want to order like a new laptop and that's above a certain budget and above a certain budget you need manager approval so that's a very typical use case for the external system interaction then there's also child workflows so you can have like one parent workflow that calls other child workflows. So this is quite useful if you want to do some kind of a workflow composition because workflows can become like very large. So maybe you have like a workflow that calls 100 different activities. Well, that can be hard to see in one file. So what you can do is you can group certain activities together if they logically make sense together as a unit and you can make that a sub workflow and then you're. parent workflow and do orchestration of these child workflows okay let's have a look under the covers of the dapper workflow engine so the blue hexagon here is your workflow application so this contains your workflow code that you've written in it and the activity code as well and maybe some other services to actually start and stop the workflow and this is the dapper side card that runs next to your application and the dapper sidecar contains the dapper workflow engine now as soon as this the system starts up there will be a communication a gpc stream set up between your workflow application and the dapper sidecar and the dapper sidecar will will detect if there are any workflows defined in your application and if there are it will also check against the state store if there are any workflows that are haven't been finished yet um so and if it is it will actually schedule the workflow uh to run again and to actually finish the workflows that haven't that have not been completed before and yeah during the the lifetime of your workflow there's going to be a lot of communication between the deeper sites where that will schedule the workflow the workflow app will do the execution of its own workflow steps of all the activities and there's a lot of interaction between the workflow engine and the state store now dapper workflow is built on top of dapper actors so dapper actors is one of the other dapper apis um it's it's not you you won't interact with these actors yourself because the dapper workflow engine will do all the interactions with the underlying actor system but i think it's useful that you know that it's built on top of the actors because you do see especially in the dapper d logs you will see that there's some actor communication going on there So there's an actor that takes care of all of the different workflows. That's the orchestrator actor. And there's an actor for each type of activity. So there's different actor types in play. And these actor types, they are stateful. So they persist their orchestration state and activity state to this event-only state store. Okay, now it's time to look at the demo. So the demo consists of a workflow application that contains... Of course, our workflow definition and activities will soon see that and that will interact with a state store, the inventory, and it will interact with the separate servers via service application, the shipping one. And this is the diagram of the validate order workflow. So it does some kind of an order validation before we actually place the order. So what happens is when we start, we started with an order payload. Then we first do an update inventory where we use the DEPR. state management API to communicate with the inventory. And if the inventory is sufficient, then we check what are our shipping providers for in this order. So this will get back an array of different shipping providers. In this case, it's hard coded to three shipping providers. And then we do an iteration for each of the shipping providers and we get the shipping cost. Then we choose which one is the cheapest. And for the cheaper shipping provider, we will register this shipment. So we will do another service application to our to our shipping service if that's all successful then we end and if this last one if this is not successful then we are going to undo this action so this is called a compensation action so this is wrapped in a try catch i will show you the source code in in a second so if this fails then we actually have a separate activity uh where we are undoing the stuff that we did here because here we already made an inventory update we're actually reducing lowering the inventory based on the order size but hey if you can't register the shipment then we need to undo that and so this is totally of course this can be anything right so this is totally up for your specific business needs okay let's look have a look at the solution file so this is the cs project file of the workflow app it's a web application dotnet 9. The only dependency that we're using here is Dapper workflow. 116, that's the latest version. Dapper 116 was released two weeks ago. Let's look at the Validate Order workflow itself. So as you can see, a workflow is just a .NET class. It inherits from a base type. It's a workflow base type where you specify the input and the output type. And this base type is from the Dapper workflow package. When you inherit, you need to uh overwrite this run async method and the run async method has like a default parameter that's the workflow context and you use the workflow context to call activities and do other things in the workflow and you have the input for this workflow so it's an order in this case okay so let me just describe what happens inside this workflow again it's already visualized in the diagram but let's have a look at this code so um so the rest of shipment result that's our final result of the of the registration we'll also we'll see that later the first thing that we use the context for is to call an activity and you call an activity by first providing the output type of the activity in this case it's an inventory result we are calling the update activity and notice that we are are using the name off so when you call activities always refer to activities by name and you provide the payload in this case it's an order item that we're using here from the order and we are awaiting this call so what happens here is that the workflow will schedule this activity but the workflow will wait until this call this activity has completed and will come back and then we have this inventory results and then the code will will continue so we can then inspect from a property of this of this return value is there sufficient stock and if there is then we will continue otherwise we just stop the workflow and we again are doing a activity call but we're now calling the get shipping providers um so this is the name this is the payload with the input but now we have another parameter set and those are the workflow task options So what you can optionally provide is a retry policy when you are calling activities. So that's what we're doing here. We didn't do it here. So in this workflow task options, we create a new workflow retry policy. And here you can use different kinds of values. In this case, I've set it to max number of attempts and it will be. will retry a maximum of three times in case it fails and in this case it will try it in a constant fashion but you can also use like exponential back off and then mention how long the maximum duration will be etc um so it's important to realize when you can use retry policies within code right and then that's very useful when you have activities that do not use any other dapper apis inside the activity and if you for instance use Debra pops up, or you use Debra certification or Debra state management. Uh, all those APIs, uh, already have built in resiliency policies based on Debra. So it's recommended that you either use the Debra resiliency policies, uh, that are built in, uh, but in case, uh, those are not available because you're calling a third party API. And that has nothing to do with Debra. Uh, then you can use this workflow. We try policy. So we have the getShippingProvider result, which then has a list of shipping providers. So what we are doing in here is we are iterating over these different shipping providers. And for each of these providers, we'll make a new request. And then we use the context again, callActivityAsync. This is the output that we expect, shippingCostResult. And we are calling the getShippingCostActivity and giving this request. But note that something different is happening now because we do not await this context call at 50 async. So what happens here is this shipping cost result task is a list of tasks shipping cost result. So what we are doing here is we are adding to a list of tasks. So what we end up here after this for each is that this is a list of tasks that you want to execute. So this is the fan out fan in pattern. And so at the end here, we have a list of these tasks that we still want to execute and the execution part happens here. So here we do await task when all, and here we provide the list of tasks. So this is both the fan in, but also the fan out what's happening here. So the Dapper workflow engine will now schedule all of these tasks and run them in parallel as much as possible, but we're also awaiting until all of them have been completed. And so the workflow will, will stop here it will wait until all of these tasks are done and once these are done we have now an array of shipping cost result and we can expect that and we are using the min buy here and using the cost parameter property from the result by choosing the cheapest shipping service all right the the next part shows how to deal with compensation action so what we have here is we have a try catch block So in the tribe block, we are doing an await call activity async, and we call the register shipping activity. And this will reach out to another service. I'll show you that soon. But in case this results in an exception, we can catch it. And these exceptions will always be wrapped in a workflow task field exception. So you can check for that. Of course, you can also check what the inner exception is here. But in case this goes wrong, we can do something else. And that's our compensation action. And in this case, I have defined another activity called undo update activity. And that will use the Dapper of State Management API again to undo the update inventory that was done at the beginning of this workflow. Okay, so that's the entire workflow. Let's have a look at an activity. So this is the definition of an activity. Again, it's just a botnet class. And then we inherit from. workflow activity again providing the input type and the output type this activity uses the dapper client to interact with a stage store so that's why we have the dapper client here so that's being injected and again you need to override the run async method we now have a workflow activity context you probably don't need to use it unless you need the workflow instance id and this is the input for the activity So what happens here is we have we use the get state on the Dapper client to actually retrieve the product inventory for this order. And we do a quantity check to see if we have enough in store. If we have enough in store, we create a new updated inventory and we save that using the Dapper client. Okay, let's have a look at another activity that will make a service invocation call using Dapper. So that's the get shipping cost activity. So here we are injecting the HTTP client. So that will actually make the call to the other service. Again, the run async method needs to be overridden, and that's what's happening here. So we use the HTTP client, and we are doing a post then to the calculate cost endpoint on the shipping service. We only see an endpoint here. We don't see an actual address of the shipping service, and that's because... Vapr will actually figure out where it is based on the name. And I'll show you in a minute where that name is defined. So this is inside the shipping service. So this is the calculated cost endpoint. I have like a bit of thread sleep here, but it is just to make sure I can run the demo smoothly, but you'll find out soon enough why. And it uses a bit of like a randomization to determine at random what the shipping cost will be. Okay, going back to the... workflow application so this is the program cs file of the workflow application and here you see that we register the http client and we use a depper client to create an http client and we provide it with one argument and that's the app id So the app ID, all Dapper applications require an ID, a unique identifier. So this is for the shipping application. So if we use the injected HTTP client, Dapper will understand how to find it because we have specified that it's the request we are making or towards this shipping app. So that's how it knows how to find it. One important thing about Dapper workflow. in what net is that you need to register your workflows and activities to the services collection so there's an add dapper workflow extension method on the service collection so here you need to register your workflow and register activities in some of the other languages you don't have to do this but in internet you have to do this so don't forget this otherwise the dapper sidecar don't doesn't have any clue that you have workflow code and we will start the workflow by calling this validate order endpoint. And when we call this, there is a Debra workflow client that's been injected and we use the schedule new workflow async method on that client to schedule our new workflow. We provide the name of the workflow. We provide an instance ID. This is optional. So if you don't provide an instance ID, Debra workflow will generate one for you. But in this case, we are reusing the order ID. as the instance id so each workflow execution has a unique instance id and you need this instance id if you want to do some other stuff with the workflow like getting information or passing it or resuming it etc that's also what you get back so in this case we provide an instance id and we are we are also getting the same instance id back and it's also important to realize because we are not getting back the result of the workflow here as an output parameter because a workflow can be long running it can take like hours or days or months or it can even be an eternal workflow that never ends so what you get back is the instance id and you use the instance id to query the workflow state later and there are some other endpoints that i that i'm using to actually make sure that i have enough stock to actually make the order All right. When you run workflows, as I mentioned, workflows are based on the Debra actors. So it's important that you actually configure your state store in such a way that actor state is set to true. Otherwise, Debra workflows will not work. Okay. I'm almost ready to run this. So what will happen now is I will use Debra CLI to run this Debra application locally. And I do this with something called multi-app run. So there's this Debra dot YAML file and that contains like the different definitions of the application that I will be running. So there's a workflow application in this folder and there's also a shipping application in this folder. So when I run the Debra command, both of these applications will be up and running. I'm also pointing to a specific resources file that contains the YAML and potentially any other resources that Debra might need like component files. Okay, what I'm going to do now is I will start the workflow app and also the shipping app. I will then make a request to this workflow app to start the workflow. But then I want to simulate that something bad happens. And something bad in this case, I will stop the workflow application. I will stop the service, the shipping application, and also the debit process. I will stop everything to indicate that there's some kind of an outage. Then I will just restart the Dapper applications again, and then we see the power of job execution. So let me first build the .NET application. So this is the shipping app that's being built. And the workflow application. Okay, so let's start running it using Dapper run dash F. So this will now read the Dapper.yaml file. So both applications will soon be up and running. All right. applications are up so now i first going to make sure i have enough inventory so i can actually make the order i am using a rest client in vs code which allows you to write your endpoints like this and so and there's also like a little button here called send request so i can click that so then i'll make a request to this endpoint slash inventory slash restock um which has a certain product id and a quantity of five so let's do that okay i'll get back two or two accepted so i have enough stock to now to order it now i'm calling the slash validate order endpoint and now my payload is an order id and choosing some some variables this is like a randomly generated thing um so i'm going to send the request but very soon then i will stop all the applications to indicate or to simulate there's an issue so making the request and i'm stopping everything Now I will scroll up a bit here in the logs because the last thing that happened in the workflow is that it reached out to the shipping application. So this is a log statement from the shipping applications, shipping service C, shipping service A, shipping service B. So because we are now in the process of getting the shipping cost. But at that moment, shipping service is down, workflow app is down. So something serious happened. All right. Now. I will restart the applications, but I'm not triggering this valid order import again, right? Because our workflow was executing. Now it's stopped, but some state is in the state store about this workflow. So Debra will actually figure out what's happening. So I'm just restarting the process and that's it. All right. So our application are running. So now we are at the same. um situation and shipping service c shipping service b shipping service a and now it will continue to shipping service a and we see that orchestration status is completed so we can also check this endpoint on the dapper sidecar by specifying this instance id so by making this request we can get the latest status of this workflow execution so i get back at 200 okay and this contains the payload of for this instance id this workflow name the validate order workflow We can see when it was created and last updated. We see that the runtime status is completed. And we also see what the input was and the output was. And so this is the power of drip execution. Something bad happened. We are only restarted the Dapper process, which if you're running on Kubernetes, that would happen like automatically. And then it got back in a good healthy state again because it just had the state was stored in the state store. It could just rehydrate the state. and continue um the other thing that i can demonstrate is the compensation action so again we can simulate an issue by returning status code 500 from the shipping application where we register the shipment so at the moment the register shipment endpoint we return some and okay 200 let's change it to return a 500 server error let's rebuild this shipping application now all right and we're going to run the app again okay and now we do have to start over because there's nothing there's no workflow running anymore so let's make sure we have enough inventory and we can now start a new workflow um and we won't disrupt it now um but now the register endpoint will throw a 500 so that activity will be retried a couple of times and those will all fail, but then a Dapper workflow engine will actually execute the compensation action. So let's make this request and let's see how this plays out now. So we're now calling the shipping application for the cost. That's fine. We now doing the retro shipment and we can already see orchestration status completed. So the workflow is completed. But now if we look at undo here, you can see that we are have called the. undo update inventory and the we have reverted the inventory update for this product to quantity five right so we've shown jubile execution pattern in in action but also how you can do compensation actions let's move to workflow management because yeah just starting workflows and getting information this is not enough you need to do more things with workflows So what you can do with DEPA workflow is besides starting an workflow instance and getting that instance state is you can also pause a workflow. So maybe you have like a very long running workflow or like it's an internal workflow and something is not working. So you need to pause it and fix something and then continue. So you can definitely do that. You can pause it. You can also resume it. You can also terminate a workflow instance. So maybe a workflow went completely wrong. You need to get rid of it. You can terminate it. So this is not terminating the application. This is just terminating the execution of a workflow instance. And finally, you can purge the state of a workflow instance. So it's important to know that by default, Dapper does not remove any state information from the state store. So if you're running many, many workflow instances, your state store will grow and grow. So you need to have like a strategy how you deal with that. And you can either use... the purge api on the workflow management part or you use some kind of other retention strategy on the state store itself to clean up this data what's also important to realize is that all of these methods require the instance id of the workflow so every all of these operations you do you only affect one workflow instance okay the final section is about workflow challenges and tips because sometimes it can be tricky to to get workflows right because it's just a slightly different thinking or programming model that you're used to so one of the big things is that workflows should be deterministic and that means running the workflow with the same output should always result with some the same input should always result in the same output and why is this well i've already shown in the animation that your workflow code will actually be executed multiple times over and over in this replay uh phase and in this replay phase um all of the inputs for an activity will be matched against what's in the state store so if there's some kind of a mismatch between what's what's happening in at one time in the workflow engine and what happens in the state store if there's a mismatch well then there's no guarantee that your workflow will be will be deterministic and will have like a good output so very practically and here's an example of a non-deterministic workflow that uses like non-deterministic code inside the workflow. So anything that needs to do with randomization or datetimes, you should avoid. So for instance, don't use like GUID new GUID in your workflow or don't use like datetime UTC now because during a replay, this will get like a new value and this as well. And if you're using these as arguments to your calling activities, this is like a recipe for disaster because this will lead to mismatches. um for these two situations there are good things right so for instance these there are different methods that you can use so the context has some additional methods so there's like a new good method on the workflow context that you can use so this is safe for replay so this you can definitely use and there's also like a helper property to get the current you to see daytime in a replay safe way as well um but yeah like i mentioned before inside your workflow code this should also only be like calls to activities um business logic so if else statements but don't do any like non-deterministic things like calling out to endpoints directly from your workflow those things all need to be wrapped in activity calls Another important part is that activity should be idempotent. And that means that running the activity multiple times should not result in any side effects. And this is because Dapper workflow guarantees at least once execution if something goes wrong in the activity. And well, you can imagine that maybe your activity contains like several lines of code and maybe failure happens at the final line of code, but all of the other statements before that haven't already been executed. So Dapper workflow will then be tried to execute your activity again so what happens when you retry those lines of code so is that safe for for to be executed again for instance if you do a database insert can you do a database insert with the same primary key for instance right probably not so maybe it's safer to first do a read and then do a write or maybe you need to do an absurd but also think about like third-party sdks right do the methods on those third-party sdks are those idempotent as well so can you invoke them multiple times without having side effects then there's workflow versioning which regardless of which which engine you use is very difficult it's very easy to introduce a breaking change in a workflow that will result again in a mismatch between the persistent data and the data at the runtime So you can imagine you made a change to your workflow. Maybe you have inserted a new activity or you're changing some types, some input types, some output types. You deploy again, but when you deploy, maybe some workflows are not completed yet. So those are called in flight. um yeah so so those are uh those are not completed but had the state is still persisted in the state store you have deployed your new workflow but now the data in the state store doesn't match anymore your new workflow model and that's a bad thing that's not something you you want so um at the moment there's nothing built into dapper workflow uh about versioning but um i think it's very likely that 117 will will include something like that but at the moment you have to do something yourself about workflow versioning and there are different solutions for instance and you can apply a version suffix to the workflow name yeah so you start with my workflow v1 and then when you want to make a change you don't make a change to workflow v1 but instead you copy that you give it a new name workflow v2 and you make all your changes in there um added the consequence of the downside of it that had your client that manages the workflow and it starts it and that stops it that needs to know about the updated name so maybe you're in control of that maybe it's the same service then it doesn't matter but if it's like a completely other service that starts or manages your workflow then it might be tricky um another strategy is that you wait until there are no more in-flight instances before like deploying the new version and maybe you are lucky that maybe there's a time window during the night when it happens that might be true but yeah the very tricky thing is you have to get insights in data is really nothing running at the moment and that's at the moment it's very tricky to to know that nothing has been started or nothing is not completed yet another option which is definitely yeah definitely safe to do is like do blue green deployments right so you keep running the old version and then you deploy a new instance add another another resource so the old one can still run or execute the in-flight instances and once those have been completed they can get rid of the of the old old instance it's definitely more complex and also costly to set up because you need to have like two active instances and you also still need to monitor when the in-flight executions when they are completed and yeah at the moment there's not really a good way to do that yet The final one is about payloads. So it's always recommended to keep like the payload small between activities because our workflow and activity inputs and outputs are always persisted to the state store and are like serialized and deserialized all the times. So keeping payload small helps you to achieve like a good performance and also doesn't take up too much storage space. So as an example here called a large payload size workflow. So let's say we have an activity that will return a very large json document by calling this document so what we have here is a very large object as an output and we're using this document as an input for the next one update document but what happened behind the scenes is that this document will be stored as an output for this activity and it will be stored as an input for this activity so it will be stored twice and so that's really not very efficient so what you should do is use of course ids as much as possible between activities but sometimes you you should also like group certain um functionality inside one activity and so in this case instead of doing like a separate get an update we do and a get and an update and save all within the same activity so we don't have to um do anything like passing big documents around inside a workflow all right um i hope i've shown you some some tips how you can actually do or actually have successful failures with with dapper workflow and if you want to know more about dapper workflow there is a lesson a track on dapper university called dapper workflow use job execution to build reliable distribute applications it's about like 45 minutes to an hour lesson that goes deep into all of the different workflow patterns that i've just shown you it's available in net java and python so give that a go if you want if you need some help in actually writing or generating code for your workflow applications, you can try out our workflow composer. It's a tool that uses a bit of AI to based on diagrams that you can either create here or upload. It will actually create real Dapper workflow applications in any language that you want. And finally, thanks for joining this session. I hope you learned something new about Dapper workflow and feel free to scan this QR code and become like a Dapper community supporter. Thanks.

Learn about implementing durable execution workflows to build reliable cloud native applications and agentic AI processes.

Applications break all the time, there could be a network issue, a cloud provider outage, or just a glitch in the matrix. But as a developer, you really need your applications to be resilient without the need to recover databases and restart services manually.

In this session, Marc Duiker (Developer Advocate, Diagrid) demonstrates how Dapr Workflow provides durable execution, which enables you to write reliable workflows as code.  Marc goes into specific workflow features, such as scheduling, sequential and parallel execution, and waiting for external events. He shows many code samples (in C#) for each of these features and will run the applications using the Dapr CLI to demonstrate their resiliency.

By the end of the session, you will have a good grasp of how durable execution with Dapr workflow  can help you build resilient applications.

Further resources:
New to Dapr Workflow? Try this free Dapr University track: https://diagrid.io/dapr-university
Diagrid blog: An in-depth guide to Dapr workflow patterns in .NET:  https://www.diagrid.io/blog/in-depth-guide-to-dapr-workflow-patterns
Diagrid Conductor Free: https://www.diagrid.io/conductor
Dapr Docs: Workflow: https://docs.dapr.io/developing-applications/building-blocks/workflow/workflow-overview/
Dapr Docs: Resiliency: https://docs.dapr.io/operations/resiliency/