Dapr University Workshop - Workflow

Hey everyone, welcome to another Dapper University workshop. In this workshop, we're going to talk about Dapper workflow and the durable execution concepts, which you can use to build like very resilient distributed applications. I'm Mark Duiker, I'm the developer advocate at Diagrid. And in my role, I create a lot of content for Dapper open source. And one of the things that I've created is these learning tracks for Dapper University. Maybe you've also joined an earlier track about the Dapper 101, which explains the most Dapper basic APIs. But in this workshop, we're going to do some hands-on learning about Dapper workflow. All right. So the first thing what we're going to do is we are, if I can copy this, yeah. I'm going to ask you to go to this website because we are going to use at Debra University please click on this link and it's the the second learning on that page here so if I scroll up here so this is the Debra University website hosted by GuyGrid and these are Debra tracks that I've created so the first one is Debra 101 we're not covering this today this was part of an earlier workshop and we're going to rerun this again in a couple of weeks but today we are going to start this one Debra workflow user execution to build reliable distribute applications So if you could click this button to start the course, you'll be asked to enter your name and your email. And it's important that you use a business email or at least an email that's not a Hotmail or Gmail or Outlook. So please use that. And so as soon as you have entered your name and your business email, you'll see another Winnote actually started. So if you then click start, you'll see this. this window, which mentions a couple of things. So first there is like a start button at the bottom right. You need to click and then this online sandbox environment will start up and it will take like, yeah, I think under two minutes and you'll see a countdown here once you start. And this environment will be active for like 90 minutes. And I think we need maybe 50 or 60 minutes to complete everything. But in case you need more time, it will be. available for for 19 minutes now what's important to realize is if that you or if you leave your computer so if your session will be idle for more than 10 minutes this online environment will stop and you can then still restart the environment but the progress that you made is then is then lost which is not a very serious thing but because all of the challenges that we're doing are not building on top of each other but it's important to to know that you are sort of restarting this whole environment and to get back to the last challenge that you're working on you can go to this progress menu here and here you can click on any challenge that you want to run because as i mentioned there's no real dependency between all of them this is this is the most logical ordering so i'm i'll be running it in this order but if somehow your session is expired and you need to continue you can start up this this environment again and just click on whatever challenge you want to continue with um all right so my environment is is ready so i can click the the start button i will wait briefly for you to also um um so here your environments are also ready um so in the meantime in this first challenge i will give like a quick summary of on the first challenge so in this first one i'll explain what your execution is and why it's beneficial for building distributed applications i will also explain that that most workflow engines implement this durable execution concept and and what workflow as code is and how dapper supports this now there's a lot of documentation available about dapper so if you at any moment need more information about dapper workflow there's this link here it's also mentioned in all of the other challenges as well so this brings you to the official dapper docs with more information about workflow And I can also recommend you to join our Dapper Discord server. There's a dedicated university channel for you to ask questions, but there's also like a dedicated workflow channel there as well. If you have more in-depth workflow questions. All right. I am going to click start here. Sorry. I hope you are ready as well. All right. So in this first challenge is more like a bit of theory about job execution. So there's no hands on part in this first challenge, but from the next challenge onwards we're going to actually run some dapper workflow applications so i think we all realize that that it's quite normal for i.t systems to fail right i mean sometimes even in the news that maybe like entire cloud providers or regions are down and it's it's not very uncommon that that applications are not available um some some part of the time and so it's also not realistic to think that all systems are available 100% of time that will never be the case And especially when you're developing distributed applications, right? Because you are communicating across like many different services and many state stores and message brokers. And it's quite likely that some of them will have some downtime and that can have an impact on your entire system. So you really need to be very thoughtful and how to handle like these temporary failures and implement retries, etc. And to a large extent, that's... already built into dapper so if you're already using dapper and that has like inbuilt built-in vtri policies which is which is great um but yeah there's actually something something more we can do to make uh doable execution possible and so ideally if there is some some major failure of our application and the whole application uh like like breaks down ideally we want to recover from that failure in an automatic fashion and so without any any manual interaction So the jubile execution concept actually guarantees that code is one to completion, even if the process that runs the code terminates, because it will actually start up another process and it will resume the code execution until it completes successfully. And how that is done is because it's actually doing a lot of state point checks to an underlying state store. And so the entire flow of your code is actually being pushed to a state store. So this principle might sound familiar, but maybe you've worked with some workflow engines. I mean, workflow engines have been around really for decades. So to automate business processes, I think it mostly began as more like visual designers. So a business user can author like a workflow using a UI. So there's always like an authoring environment. There's also a workflow engine. component to it to actually run the workflow and there's all in this usually also a stage door behind it to actually persist the state of the workflow so it can resume in case it fails and so these three components offering environment engine and state store those are three crucial components of any workflow system and yeah workflows are really ideal to handle like long-running processes right so maybe you are doing some kind of an approval process like an order approval process that involves some some manual steps in case you have like a like a manager approval um so these systems can run maybe for like minutes for days for months for years or even like like indefinite so yeah workflows are you for these very long-running business processes um some systems or yeah most of these systems started out with like visual designers but since the last couple of years a lot of these workflow systems are based on workflow as code which means you're not using a ui to design or to create your workflow but you're actually writing your workflow in code and i think that that's a great step because well i come from a developer background i like code and i definitely like to write these workflows as code because well it's part of version control so all of the changes will be reviewed in pool request hopefully and you can also unit test your your work which is a great thing and most of these official tools don't have that of course it can be a downside that for flow as code typically doesn't have a like a visual representation so they're like pros and cons to to workflows code solutions now um Since Dapper 1.13 or even Dapper version 1.12, there is like a workflow API. So there's a built-in workflow engine in the Dapper sidecar. And recently, since I think 1.15, that became like a stable API. So it's been on development for about two years. And at the moment, you can use all of the commonly used languages to use Dapper with. like .NET for to author these workflows you can use Java, Python, JavaScript and Go. In this lesson however only .NET and Python are available. So if you're .NET or Python developer you're free to choose these two languages. I'm still working on the other languages to be added to this workflow track. So in this diagram here, it's just a representation of what lives where. So you have a workflow application that you write. So you write your business logic in your workflow application by using workflows. And the workflow is built up of small tasks. And those are called activities. And so that's all in code in your application. And your application communicates with the Debra sidecar. And the Debra sidecar contains the workflow engine. So as soon as you start or schedule a new workflow, that will be communicated to the Dapper sidecar and that will actually initiate the execution and communicate back. So, okay, now start with execution of this workflow. What the Dapper workflow engine also does, it will communicate all of these state changes of the workflow to a state store. So all of the changes. that we made are persisted to a state store i've got an animation of this in the next challenge so let's let's have a look at that okay so i will continue with the the second challenge um that shows like the basics of how you can actually offer a workflow so um i'm also showing you how workflows are started and how to get the status of a workflow instance and how the workflow engine actually persists the workflow state all right um So a workflow, like I mentioned, is like a collection of smaller tasks called activities, but also some business logic. So this is like a very small example of how it visually looks like. Your workflow contains like activity calls, but also some if statements or switch statements based on the output of an activity, choose a different path in your workflow. So I have like a very basic workflow demo in this in this folder. I'll first expand this to show it visually. So what happens in this basic workflow is we will start the workflow with an input argument with a text string named one. Then we run an activity that just simply appends two to this input string. So at the end of activity one, we have one, two. And then we run another activity that appends three to the input. So the output of the activity is then one, two, three. And that's also the output of our workflow then. All right. I won't run through the .NET examples here, but as you can see, you can either choose either the .NET explanation or the Python explanation. So I'll run through the .NET workflow. And here in this area, you can actually browse through the code. And these are all terminals. And we're going to run this. We're going to run some commands in these terminals, actually run the application, make some HTTP calls, and finally check where the state lives. So if I open the basic workflow file here, so I'm, maybe I should make this also a bit bigger. So if I open this basic workflow file here, so this is a C Sharp project. I have a basic workflow class here. So I show workflows in .NET C Sharp are just classes. And you are using a Debra workflow package in this case. Since you are offering workflows in a specific language, you do need the specific language package, the workflow package, because that contains some base types that you'll be using. So if you're creating a workflow, you need to use the workflow base type. So you inherit from that and then you specify the input type of the workflow and the output type of the workflow. So if you inherit from workflow, then you need to implement or override the run async method. And that always said the first argument of the run async method always has the workflow context. So this is something that's provided by the Dapper workflow engine. And on this context, there are like properties and methods that you can use to actually call different activities, create timers, wait for external events, etc. I'll mention all of these later. And the second argument is your input for the workflow. Now the output, everything in workflows is asynchronous. So the output is async task of string. I've configured that to be my output. Output types can be anything, right? So in this case, I'm using a simple type. It can also be a complex type as long as the output is serializable. So this is a very small workflow of just two activities. And here you can see we are using the context and then we're using the call activity async method on the context. We specify the output type that we expect back. then we specify the name of the activity you can also use a string but it's a little fragile to directly use strings so that's why i'm using the built-in name of function in c sharp to give me back the string representation of the activity one and i'm specifying the input for this activity now note that i'm awaiting this call so what happens when we start this workflow then as soon as we use this this is called async API we are sending a signal to the network for engine from now police kill this activity one talk and what happens then the control flow of the workflow will stop and the depra workflow engine will signal to schedule this activity our application will actually run this activity and it will then as soon as the activity is completed then it will give control back to the workflow I'll come back to this later. So what happens as soon as you have completed the activity, we have a result here. And as you can see, we are using this result here as an input for the other activity. And then when we call it 52, we use that result as the output of the workflow. Now, if you look at an example of an activity, let's just use this one. It's basically the same. the same setup we have like a regular class we inherit from the base type workflow activity we specify the input and output types and again we have to override a run async method in this case we have a workflow activity context you probably don't need it that much because the only thing that's on this context is the instance id of a workflow so as soon as you start a workflow It always have like an identifier that's called the instance ID and you need this instance ID to be able to do other operations on the workflow, like getting the status information or do anything other related to the management of the workflow. But that's part of the one of the last sections. And the second argument is the input of this activity. So nothing really happens here. We just do a control right line of this activity received to inputs just to make sure we see something in the logs here. And we give back a result very simply append to this input. The same thing happens in activity two, but then there's like a difference. Then there's three appended to the inputs. And the final thing that you need to know about authoring workflows is that you need to register them in your startup code. So here we can see the services collection and there is an add dapper workflow extension method that you need to use. And there are two methods on there, register workflow. And you need to specify the type of your workflow that you have written in your workflow application and you need to register the activities. So this is something that's specific to .NET and that is not in Python. So if you're using the Python version. you don't have to do this registration of the workflow and activities that's been dealt with by the sdk itself but in case of.net you actually do need to do this registration of the workflows and the activities now this is a web application and it also has an endpoint start input and we are injecting the dapper workflow client and the dapper workflow client is used for the management operations of a workflow and the one with that we're using is we are scheduling a new workflow. So we are awaiting this call. We provide the name of the workflow and we provide the input. And what we get back is we get back the instance ID. So we get back the identifier of the workflow that's been scheduled. So we don't get back the output of the workflow immediately because it can potentially be a very long running workflow that can take days, weeks, months, or it might never end. So that's why we get back the instance ID so we can later. provide this instance ID to retrieve information from this workflow. Okay, let's see if I covered everything here. Oh yeah, this is still important. So the workflow code itself is just like using the context to call activities and have some if else statements. And these are not in this example, but that will be in later examples. But it's important that you limit your code to just call activities or calling child workflows even. get back to that in a later challenge and some business logic and nothing else. What you do in activities, in activities you typically call like other external APIs. So maybe you want to store some data, you execute that in an activity call and you're calling a third party API to another SaaS service. You'll do that in an activity API. So all of the things that I just mentioned, don't put them in a workflow. the workflow should only contain calls to activities and some business logic okay so now i'm going to run the.net application so i first need to navigate to the right folder here and like i mentioned before as you can see it also here in the label this this learning track just clones the quick starts folder and inside the tutorials quick start workflow c sharp i'm navigating to these different folders for all the different exercises So I'm also going to point the terminal to the fundamentals folder now. And I will now build this solution. And the first time it's probably going to take a bit longer because it also needs to do a .NET restore. Or in case in Python, you do need to do like an installation of your requirements file. So initially you need to download all of your dependencies. So that probably takes some time. But on the other challenges, it will be a bit faster because you've already downloaded all of these. Okay, so it built my small solution. And now I'm going to use Dapper Multi App Run to start my solution. Now, if you're not familiar with Dapper Multi App Run, so that's this command that I'm running, Dapper Run dash F and then dot. What this means is I'm going to run Dapper, but I'm specifying a file. So that's what the dash F means. And the dot means where's the location of the file now. there's in the root here there's a debra.yaml file and this contains a definition of the applications that i'm going to run in this case it's just one application but multi-up run is really useful when you want to run and start up multiple application at the same time because this is basically like an array of different applications in this case it's just one application and i'm also pointing to a resources path and this contains a yaml file that describes what kind of underlying state store i'm using but i'm postponing that for just a few minutes i'm coming back to this later now let's run this command and it will create a new application id it will go to this folder it's going to run dotnet one to actually run my dotnet application all right that's all of these um commands i am executing in the dapper cli folder so this now shows some some debug uh some info statements from my application that's been starting and also the dapper process that's been starting and it will take a couple of seconds before everything is up and running and that's because this online sandbox environment is quite light so it's not like super performance um as soon as you see like placement table updated if you see something like that where was it here that that's the moment it's actually ready all right So now our application is up and running. So now the application is up, but the workflow has not been started yet, right? So we first need to, now we are going to call this start HTTP endpoint to start the application. So I'm going to make a curl request, make this a bit bigger. Head to this local host endpoint and start, I'm going to provide one as my input argument. So I'm going to do that. This will be in my curl window. All right, so it went really fast. um have we see we go back a two or two accepted and we got back uh in the location header there's an id and this is the instance id of the workflow that's been started now we can already see in the debris ali we know that workflow completed with status orchestration on the score status completed so we already know that the workflow actually completed if we scroll up a bit we also see some some application logs that the activities have been called Okay, so we got this. Let's scroll a bit further. Now, what we can do is now use this instance ID to actually get back the instance or to get back the status of this workflow. But I want to actually prevent us from copy-pasting this ID out of this header into something else. So what I did is I could actually execute this curl command. and that actually extracts the instance id from the location header and puts it in an environment variable so i'm going to run the workflow again it will create a new id and this new id will be captured in this instance id environment variable and we can use the environment variable in subsequent interactions when you want to get the actual state so i'm running this again in the curl window all right so we can already see in the logs it's completed but if we echo out the instance id So we get back like three, two, three, one. So that's the start of our instance ID. And now we can use the get commands of the workflow management HTTP API to get the status of this workflow. So I'm going to make another curl request now. And this is not going to my workflow application, but I'm targeting this directly to the Dapper process that runs next to my workflow application. So it's still local host because Debra always runs next to your application, but it's a different port. So this is the port where Debra runs. Then it's the version one of workflow because workflow is now a stable API. And then it's always workflow slash Debra. And then you provide the instance ID. And in this case, it's an environment variable. So I'm going to run this in my girl window. It's a bit squished here. Let me make this a bit bigger. Oh, it's still switched. So here is the output of this get command. So what you get back is this JSON object, which contains the instance ID. It contains the name of the workflow. You can see when it's created, when the workflow was last updated. You'll see the runtime status. It's completed. And it also contains properties, which usually contains the input one and the output one, one, two. I think it's missing the three, I think. It's been cut off. All right, so it is important to realize that whenever you execute a workflow, you never get back to the result because it's asynchronous. You always need to do a get command to get the result. OK, we can actually stop this and I'll quickly check again if there are any questions. But first, let me finalize this chapter. I want to explain a bit about workflow state and replay. So as soon as you as soon as you start a workflow, all of the state changes will be persisted to a state store. So I've created this animation to give like an indication of what's going on. So if a client has scheduled a new workflow instance, there will be a lot of communication between the workflow application, the Dapper workflow engine and the state store. So I'm going to play this. I'm not sure how well this full actually looks in the... webinar environment but you can always play this for yourself in your own environment of course so we schedule a new a new workflow instance so the workflow is started and then we start a new activity but and then also the inputs and outputs of the activity are then assisted to the state store when the activity one is done it won't immediately start 52 but it will replay from the top until we read the information from the state store because activity one was already completed then it will execute activity two and again it will like persisted all of the inputs and outputs to the state store once activity two is completed again it will replay from the top it will get the information from activity one and two from the state store and then it will do it 53 and once that is done it will again do a final replay and then it finds that all the executed all the activities have been executed it reach everything from the state store and then your workflow will be finished then it also store the output in the state store so there's a lot of io happening um when you actually are running workflows but that's intentional because we want to do all of these um yeah stores to the state because at any moment and the application might fail so we want to do all of these checkpointing to make sure that we can actually easily get back in a good state um now we didn't actually see where our state was also stored because yeah with dapper you can configure any uh dapper compatible state store and that's compatible actually with actors to be the stage for workflows and that's because dapper workflow is built on top of the dapper actor system so dapper actors is like a separate api that you can use to to write actors and but deborah is using that internally to store the state of the actors and to store the state of the workflow but also of the individual activities So in this case, the state that I'm using is the default state store. When you install the Debra CLI locally, you get a Redis container. So I'm using that as the state store. You only need to make sure that whatever state store you use, you need to check if it's compatible as an actor state store, and then you need to set the actor state store field to true. So that's one thing to take into account. And in case we are using Redis now, I can actually look up the keys now in Redis. that contain the data for for this workflow so I'm going to do a query in Redis where I want to list the keys that that contain this yeah this string so basic and then double pipe and then dapper internal default basic because that's the name of the workflow and then dot workflow so if I run this in the Redis terminal we get back a lot of information here because there are actually a lot of keys that that match this and that's because the workflow is replaying a couple of times so for each instance so here is instance three two three and there are a lot of like history checkpoints uh going on that contain the execution state for this workflow now uh so it now even although you can see where dapper stores its state and and yeah where this is located you should not actually interact with the state directly So you should leave it up to Debra how this state is being managed. And there is a workflow management that you should use if you want to clean this up or if you want to get rid of it. So it's not recommended to interact with this state in any other way than via the Debra API. All right, let's move on to the next challenge. And I can be very quick about this because this is about the first workflow pattern. doing workflow as code there are different patterns that you can that you can write and the first one is actually very familiar to what we just saw because it's about task chaining and you use task chaining when you want to enforce a specific order between all the activities and that's because there is likely a dependency between the activities so maybe you first need to excuse the 51 before you can continue to activity two because probably you need the output of 51 as an input for 52. or maybe activity one does something to an external system and that needs to be done first before you can do this other activity which also does something to an external system which has a dependency what happened in activity one so have a task chaining the the order is of importance and that's why you create this chain of activities I'll just run quickly through this sample since it's very similar to the first one. So I'm starting this with just one string input, this, and then I'm calling three activities which just simply append a word to make like a sentence and the output is this is task chaining. I will just show the program file again. So we are registering our chaining workflow. We are registering our activities. I'm calling this start endpoint and we are scheduling. this chaining workflow and we're giving it input of this and i will only show the workflow itself but not show the activities because they don't contain that much um and so here here is the run async method again and this is the chain of the three activities where the output of the activity is always used as an input for the next one and the same is true for number two and then the third one is actually our workflow result okay let's run this Let's first move to the right folder, task chaining folder. Let's build the solution, but should be quicker now because we've downloaded the dependencies on this environment. And once this is done, yeah, it's done. Then we're going to run the application again. All right. And if I now immediately start the workflow, so to call this endpoint and capture the instance ID and then the variable. So let's run this in the curl window. So when I'm very quick, you'll see this error processing operation, uh, Debra built an actor, not found retries. So you can actually ignore that because we are actually sort of too soon. Uh, Debra has not fully initialized yet because it's just a very slow environment. Uh, but Debra is actually like, uh, using it, it's own retry policies to actually get to a healthy state. And then finally you can see that our workflow is executing. We can see the application logs from all of the activities. And we can also see that the, uh status is completed here in in the logs and again you can also use the other get api to actually call the sidecar to get the same information that the workflow is completed okay so this is the first workflow pattern task chaining most basic one probably the one that you will also most often use all right we go to the next um pattern and that's a fan out fan in so the fan out fan in uh pattern is really useful when there is no dependency between the activities you want to call so in this overview we will start a workflow but then we will immediately call three activities and we'll try to execute them like independently so as close to each other as possible and so you could say it's sort of parallel it's not guaranteed to be parallel but it would be um yeah as much parallel as possible and the nice thing about the fan out fan in pattern is first the fan out happens so you all of these activities are called and i'm showing three here but these can also be thousand activities right but then you can instruct the workflow to wait until all of these activities have been completed or until the first activity has been completed so you can do like a when all or when any in the workflow and and that is the fan in part again so then you can actually use some aggregation over the output of all these activities so that's quite powerful so the sample i'm going to run in this track is this i'm going to start a workflow which is an array of words and i want to find which word is the shortest one in this array it's a bit of a silly small example but it does prove the the mechanism behind it so what i'm going to do is for each of these words in the array i'm going to call the same activity and with fan and if an infinite out you don't necessarily always need to call the same activity that those can also be like different activities but in this case for every word i'm going to run the same activity which in this case simply returns the length of the word and so the output here um so the aggregation at the end when they have all been executed is okay give me the word that's the smallest and that should be is so quite quite a basic example but it's i think it's a good example to see how this pattern works. So let's have a look at the workflow. That's this one here. So this is the, how to run a sync method that needs to be overridden. We first do like a short like input check if there's actually any valid input. And what happens then is we create first a new task, sorry, a list of tasks of word length. And this is the output type of the activity that, of the activities that will be called. So once we have this new list, we will do a for each loop over all of the individual strings in the input array. And here we are using the context to call the activity get word length. And that will give us back a word length object. And this is the input that we're providing. But notice that we are not using await here. So we're actually not awaiting this call. We are just. merely adding this this task to call this activity to this list of tasks and this is the place where the magic happens so here we do await and do a task when all and this is then the list of tasks so this is the moment where the workflow application signals to to the depo workflow okay now go and schedule all these individual tasks and then the workflow application will actually run all these individual tasks And at a certain moment, all of these tasks are ready, all these activities. And then the control comes back to the workflow application and we can actually continue with the next line. And that's actually checking the results and then ordering them by length and then choosing the first one, which gives us the shortest word. Okay, so let's run this. I'm skipping over the program.cs file because there's nothing really new in here. first need to run the application here we go so let's move to the right folder i am building the fan out for in solution okay it's built running the solution there we go and now i can actually start it so again so here you can see that i'm starting it with a payload of an array of strings here and again we're capturing the instance id so we can actually use the get request to get back to status so uh let me run this again we were a bit a bit too fast here so yeah now it's actually running and we can see here in the application logs that the get worth length activity is called with each of the individual inputs we can also see that the workflow is actually completed so let's see if the result is actually the word is by running again this get command and specifying the instance id And so the input is which word is the shortest and the data output is actually is. Okay. So this is a quick example of how to use the fan outfitting pattern. All right. So the next pattern is the workflow monitor pattern. And that's a very useful pattern if you want to execute like a recurring task. So maybe you will have a daily or a nightly job running that cleans up some cloud resources, for instance. so what you do there in a schematic way is you first want an activity to do a cleanup job or maybe you're checking a status of something and then you want to wait for a certain moment of time so you instruct the workflow to create a timer and when the timer expires you actually instruct the workflow to continue as a new instance Now, this is something different than the replay behavior of a workflow. The replay behavior happens when something is actually continuously running and executing. But continue as new means you start this instance of a workflow as a new fresh instance and you can forget about all of the historical executions. It's basically like a do loop is actually what's happening here. In the sample application that I'm going to run, we have this situation. So we start with a numerical input. We start with zero. Then we have a check status activity, which simulates getting a status from some kind of an online resource. Then we have an if statement to check, is this status? Is it ready or not? If it is ready, we just immediately complete. So we stop the workflow. and it then gives back an output and then checks okay how many times it actually through the cycle to actually come to this status and if it's not ready then we are creating a timer we wait for one second in this case not to wait too long and we actually increment this this status counter and we continue as a new instance and we provide this counter as the input for the new instance because we're also starting with a numerical input here so we can actually update that input and provide that when we continue as a new instance all right let's have a look at the workflow now so this is the monitor one so here we have a run a sync method again and we have the integer as our input and so the first thing is yeah this is usual we just call on activity and it's called stack check status we provide the counter as as an input and we get back a status so if the status is not ready then we are using the context to create a timer so this is a method on the workflow context and we provide a time span in this case just one second and we then also increment the counter and then we use the continuous new method to uh on the on the context and providing that updated counter If the state is ready, we just skip this part and just return the workflow. Now, if you look at the activity here, there's not a lot happening here, but we do a console right line and that also then includes the input. So we can also, we can just keep track of how many times this has been executed. And we then use like a bit of randomization to sort of randomly determines if this state is ready or not, just to make it a bit more interesting to run this demo. all right so this is our application startup so we have a post endpoint that we go to call this is start slash counter and we're going to start with zero and this is going to start our monitor workflow all right let's the application that's built the application all right and let's run it using multi-app run again so our application is starting up and debra is starting up so now let's call this workflow and so we're calling start slash zero so we're starting at zero call it in a curl window again it complains that uh actually it's not ready yet but it will be soon and once it starts we actually will see some console logs yeah it's now ready yeah and here we see that the workflow is like restarting right so it's already and it's in this case it's restarted like until we have received input 4. Okay, and now we actually see the status orchestration status completed. So if we do a get now to check the status, execute this scroll command and we see that the output is now status healthy after checking four times. All right, this is like a very basic example of using the monitor pattern, but yeah, it's a very powerful thing to keep doing things on there. recurring thing especially if you include like some some business project there stop this and you can move on to the next one yeah so this is also a very nice one so the next pattern is the external system interaction pattern so this pattern is very useful if you want to pause your workflow if you want to introduce a waiting step in your workflow and you want to have another event controlling the next flow And so in this case, it can be very useful for like an approval workflow. So maybe you want to order like an expensive laptop and you need like manager approval for this. So initially the steps like before this wait can be like, okay, process the order and determine what the total price is. And if the price is above a certain threshold, then notify a manager and then the manager will. review this order and it will use their application to send back an event to this workflow instance and we can then inspect this event the payload of the event and then we can continue with the workflow but we can actually make a decision based on the payload of the event as or something can be approved or not approved or you know whatever your business logic is so we can actually yeah determine what kind of branch in the workflow we'll be following based on the event that's coming in to your workflow So in this case, I do have like an example of like an order approval workflow. So I'm actually having an input of like an order with rubber duckies, ordering 100 of them and a total price is like 500. So my first if statement in my code is the price over 250. In case yes, then I'm waiting for an external event. Because in this case, I'm assuming that my manager is already notified and that an approval event is coming in. Once the approval event comes in, then I'm checking the payload of the event. Is it approved? Yes, I can process the order. And then I'll send a notification. And then I'm stopping in case the order is not approved. I'm not doing anything with the processing the order. I'm just sending a notification that it's not approved. And in case the price is not over 250, then I'm like automatically processing it and sending a notification. So let's have a look at the code here. So this is our external events workflow. Give it a bit more space. All right. So again, I have my run async method. So now I'm using like a POCO, like an actual .NET object as my input, which is like a serializable object. I already have like a default. approval status that i'll be using later so my default is actually says um i'm already approving it so that's my my default um i'm first checking if my order has a total price of over 250 right so this is my first if statement um and then i have like a try catch block and in the try i am waiting for an external event and you are always need to specify the a type of the event which is in this case approval status and you're all always need to specify an event name in this case i'm expecting an event with the name of approval event now the second argument is a timeout argument that is optional but i am definitely suggesting you should always use this because if you don't specify a timeout event and you're just your workflow is just waiting for the event but what if that event never comes in right so you always need like a sort of a fallback solution in case the event does not arrive so i i think business-wise it's always good to have like a backup solution to always specify a timeout event but if it never arrives well in this case you need to do something else so in this case if an event does not arrive and a timeout event is triggered we actually receive a task cancelled exception from uh from the dapper workflow so in this case the timeout is triggered and i prepare a message and i'm just calling the send notification activity with this message but of course you're of course it depends on your business logic what you want to do here right but in this case i'm just simply calling an activity and just sending a notification um so if the catch is not triggered that means that we have received an approval status event so i'm checking the payload if it is approved if it is approved i'm calling my next activity and it's processing the order and then finally i am creating the final message that's based on the is approved so it's either order is approved or order is rejected and i'm sending a notification with this notification message so again it's a bit of a simplified solution but can already see that we're already introducing more and more business logic with if statements and try catch blocks and that's very typical for for workflows code systems um let's see so we're recording this start endpoint and we are getting an order you get the deborah workflow client injected again so i'm calling this external event workflow we are now specifying the instance id of the workflow so this is something new so so this is like an optional argument if you don't specify an instance id that will generate one for you but in this case we already have an id and that's the order id and i'd like to use that as my instance id so i'm specifying that and i'm specifying the order and that's it okay so let's run this move into the right folder let's build the solution make this a bit bigger again okay and now let's run the solution So now I need to do like two things. So the first thing is I actually will start the workflow. And as you can see by this payload here, so here's the data. So this is an order. So this is the order ID. There's a description. There's a quantity and there's a total price. As you can see that the total price is 500. So I'll be triggering that if statements that actually need like an, so I'm actually will be waiting for an external event. And there was a timeout of, let me check, I think it was 120 seconds. Let me move up here. Yes, so we're waiting for this event and we have a timeout for two minutes. So what I'm going to do is I'm going to start this workflow and then within the two minutes, I will actually raise an event to this workflow, which in this case is an approved event. So let me first... Start this. Okay, so I'm going to run this and I don't need to capture the instance ID now because I'm actually providing the instance ID because that's the ID of my order ID. So that's why I'm not capturing anything in here. Okay, so now the workflow is here, but now it's actually waiting. So now I will send like an external event and in this case I am sending the external events directly to the Dapper sidecar. And so this is not what you would do in production. In production, you would actually wrap this call or use the Dapper SDK in another endpoint. But in this case, I'm calling the Dapper sidecar. So I do this endpoint. So this is my Dapper sidecar address, the version one of the workflows, Dapper. And then this is the order ID, which is also now R. instance id and then it's slash raise event and then this is the name of the event and then i'm also specifying a a json payload as well so this is the data that i'm sending to the event is approved true let's see if i'm still uh within the two minutes um i'm too late so we actually we get back an error code fields to to raise the event on workflow, field to raise the event on workflow, workflow not registered. So I think this actually means that I was like too slow and this already completed because I was just talking too much. So let's actually run this again and then I'll be a bit quicker. So we're going to start the workflow. All right. Oh, no, I think something else is going happening now. Something's failing. OK, in this case, something else will be failing. restart my application okay maybe the uh the affair the environment had a glitch or something so restarting my application now i am going to make the call to the workflow again all right so that's all good and then i am raising that event okay i'm raising the event uh yeah i think it's worked because now we get back orchestration status completed so let's actually do the get the request to make sure what the thing is oh it still says still says too late timed out okay well i'll just continue for for time's sake but i think you get the picture on how to work with this external events flow let's move on um so Another pattern that you can do is called child workflows. So what you actually can do, you can have like a parent workflow and the parent workflow, you can actually call child workflows. So you can actually nest your workflows. Now, why would you do this? Well, maybe you have like a piece of reusable gold, some activity steps that you always run in a specific order, and maybe you want to reuse that across many different workflows. So it's like you do some composition of your workflow. So maybe you do this also for like. to be able to test it more easily. So instead of having one gigantic workflow of 100 steps, maybe you can have like 10 smaller workflows of each 10 steps. In this case, I am calling a workflow that receives or that I start with an array of two items. And then for each item in the array, I'm calling the same workflow. So in this case, I'm just executing two child workflows of the same type and then aggregating the result and then ending it. Let me quickly show it because we are already a bit short on time, I see. So this is my parent workflow and I receive a string array as an input. I create the list of tasks of the output that I have and then I'm iterating over the inputs and I now use the context to use this method to call child workflow async. So instead of call activity async, I'm now calling child workflow async. So this list of tasks now has all of these. task which define to call child workflows and then i'm using the same fan outfit in pattern to actually do a weight task when all and then providing all of the tasks of the child workflow um due to the time i am not running this because i still have some other challenges i want to go through so this is an important one because this is about um resiliency in workflows and compensation actions so one of the additional things you can do with that workflow which is already very resilient by default because i have the whole process collapses you can actually restart the process and debra will sort of selfie will continue the workflow but for instance what you can do when you call an activity you can provide some resiliency policy that will retry the activity in case it fails so that looks like this here we go this one So this workflow here, I'm specifying a default activity retry options. So that is of the type is workflow task options. And in those options is a retry policy property. And this is a new workflow retry policy. And that has like several configurable options. In this case, it's like a constant retry of three times. And the first item that I want to retry this is after two seconds. But you can also configure like exponential retries as well. and so this is now a new default activity retry options and i can provide this as my final argument when i'm calling activities so i'm first specifying reactivity name the activity input and then i specified retry options so another thing that is happening in this in this workflow is in this case i have to strike catch block and i am calling activity called division Now, what if this workflow will fail? Well, then first the retry options will be executed. So this will be executed for maximum three retries. If it then still fails, then there'll be an exception and that exception will be wrapped in a workflow task failed exception. So I'm catching this workflow task failed exception. I can inspect the failure details as you can. yeah determine in detail what the actual underlying exception is and in this case um i'm forcing it to actually be like a division by zero which actually in dotnet results in an overflow exception and if that's the case i am running a different activity so this is called a compensation action so i can actually provide like an undo action which is just a new activity that i'm writing myself to undo one of the earlier steps so my first step that i'm doing here is actually doing a minus one so i'm doing some kind of calculation here i'm subtracting one and my compensation action here is actually do a plus one here so again this is a very overly simplified solution but i think you can you understand how to actually apply this to your own to your own business logic um another thing that we haven't seen yet there's a method called set custom status so that is a method that you can run on the workflow context so you can always provide a bit of extra custom status to your workflow instance as you can also for if you want to keep track or add some more debugging logic or something you can actually provide this with not just a string but with any object that you want to provide a bit of additional status okay i think there are like two more to go um yeah so this is a more realistic uh challenge i'm not going to run them because i think this is nice if you just explore this on your own This is a more realistic example that actually interacts with like a state store. It also interacts with another application, a shipping application. And so it uses the fan out fan in pattern, then has some business logic to process a payment. So this is, of course, still simulated. Again, it interacts with a state store using the Dapper API. It then uses Dapper PubSub to communicate with the application. So this interaction was with certification, but now it's using PubSub. And the shipping application is then also using PubSub to actually communicate back to the workflow application and then raises an event because the workflow will be waiting until it gets back a confirmation about this registration. And then we determine if it was successful. If not, we apply a compensation action to actually reimburse the customer because it's paid for it here. So we reimburse it. And if it is successful, then we immediately end. This is definitely a more complete example for interactions with multiple applications. Definitely have a look at it on your own. This part is definitely important because the Dapper workflow is of course about authoring workflows as code, but you also need to manage these workflows. The Dapper has a workflow management API, and you can not just start the workflows. get some information, get the status about them, but you can also suspend a workflow as it means like I'm pausing a workflow. You can also then resume a past workflow. You can also terminate a workflow. So maybe you have like a workflow which is completely out of control or maybe it is stuck or doing some weird things. Then you can terminate it so it stops executing. And you can also purge a workflow instance. So purging means we are removing the state from the underlying state store. and this is very important because dapper workflow does not do automatic cleanup of your workflow state so as soon as you start using workflows and you'll just adding state continuously to your state store and yeah you are responsible yourself to to deal with the state management so you can do that via this this purge endpoint and what you should know is that all these interactions are based on a workflow instance so you can suspend or resume terminator purge just one workflow instance so at the moment there is no way to do like a batch terminator to a batch purge that's that's something that is definitely needed in the future to quickly purge like historical data for instance so at the moment it's very important when you run that production and to always make sure to store your workflow instances somewhere else to make sure you can access them and then purchase them efficiently later. This also comes with like a demo application which actually runs like an indefinite workflow, but given the time I am not running that now. The final part that's about some challenges that there are with workflows. So the first challenge that you have is your workflow code needs to be deterministic. And what does it mean? Well, your workflow code will be run many many times due to the replay nature right so it won't be just run once but it will be running multiple times and all of the code that's in here should be deterministic meaning for all of the same inputs it should always result in the same outputs now some examples of non-deterministic code is for instance directly using GUID new GUID in your workflow because for every replay you'll get a new order id in this case and you if you're using this order id as an input argument for calling this activity all of the input arguments are positioned to the state store and all of the output arguments are stored as well and they're also looked up in in this way so you can easily get a mismatch between um state from the from the state store and uh state at the at the run time the same is true for daytime you to see now and so timestamps should not use timestamps in your workflow so luckily most of the depra sdk's has some helper methods for this so for instance there is a context new guids for the dotnet one and that that's not yet on the python sdk i think um there's also a current you to see date time um property on the context and that is available in the python one so you can use that instead of like using the daytime stems for anything else that non-deterministic put that in an activity because activity code can contain anything and that's because the inputs and outputs of these activities are are stored another important thing is that activities should be idempotent meaning that whenever you execute an activity if you execute it multiple times it should not have any like unwanted side effects and that's because dapper guarantees and at least once execution in case an activity fails so an activity could be run twice but yeah it means that in case activity fails then maybe some code is already executed correctly in your activity so you really need to think about all of the apis that you use in an activity are those apis idempotent right so for instance if you insert a record in a sql store in a postgres database can you run that insert twice or will you get issues so maybe it's safer to first do a read and then do a write or do an absurd instead of an insert those kind of things another thing to think about is check the payload size of your workflows and activities so as you saw in the in the video and i've also mentioned it there's a lot of io happening between your workflow application between the sidecar and the state store so it's important to to make your payload as small as possible to make so that the serialization deserialization is like optimal so and this is like a an example how not to do it so for instance if you are calling an activity to retrieve a very large json document and you're then passing that large JSON document from one activity to the other one. And that means that first this document is stored, saved in the state store as an output argument, but also as an input argument for the next one. So this document is stored twice. So in this case, it's better to just combine those two activities into one activity and you retrieve the large payload, you update it and you save it all in the same activity before continuing. The final tip that I have is related to versioning. So versioning workflows and versioning workflow as code in specific is a tricky thing. I always recommend to use like update a version number as the name of the workflow. As you see here now, I have versioning workflow one and here I have like versioning workflow two. So this is a safe way of doing this. And then why is that? Well, let's say you have a workflow, your first iteration is running. You have maybe there are like long running workflows. So maybe the execution takes a couple of days before they complete. You have like updated the workflow and maybe there's like a breaking change, which is likely. so you deploy that new version but there are still workflows in flight meaning they have not been completed yet because they're like halfway through the execution now you have a new workflow definition a new model that's running but you still have the old state in the state store and this state is not compatible with the new version anymore so you'll run into issues when you run this updated version and so since everything is always persisted with this key that includes the workflow name, it's always recommended that you then just include a version number in this workflow name. You never have issues with multiple workflows because if you deploy this new version, you still have the old version in there as well. You definitely use a version number, but you also keep the old version in your solution. Existing workflows can still execute this one and new workflows. can start using this this new one this does mean that your client which is responsible for starting this new workflow and your program cs file is of course updated to use this new version

In this hands-on workshop, you'll learn about how you can use Dapr Workflow to improve the reliability of distributed applications. Link to course: https://www.diagrid.io/dapr-university

We'll cover:
- What durable execution is
- How Dapr Workflow works
- How to apply workflow patterns, such as task chaining, fan-out/fan-in, monitor, external system interaction, and child workflows
- How to handle errors and retries
- How to use the workflow management API
- How to work with workflow limitations

Supported languages: .NET, Python