Failure is not an option: durable execution + Dapr = 🚀 with Marc Duiker

So this is really, yeah, it is part of the problem that also our systems are becoming so complex. And this is nothing new, really, because in 1994, two people, Peter Deutsch and James Gosling, created these fallacies of distributed computing. Did you ever hear about these fallacies? Yeah, some people aren't familiar with this. yeah so it's it's a set of false assumptions programmers new to distributed applications often make yeah so all of these assumptions are of course like incorrect but people new to distributing yeah they they simply don't know or they don't think about these things and this is not a theoretical session so i'm not going to cover all of these eight points i'm definitely going to highlight some of the links i've shared here here below especially the bottom one is like a youtube playlist from odin from particular software who really very clearly explains all of these fallacies. So definitely have a look at that. It's also very important to know that not all failures are the same. So in this case, I am looking at technical issues and not more like operational or other type of issues. But when it comes to technical issues, you have transient failures, which are temporary and usually resolve themselves. And you have permanent failures. And those are, indeed, as the name says, permanent, they are not temporary, they don't resolve themselves. But when they are transient, meaning they are resolving themselves, of course, there's nothing, there's no magic involved there. They are transient because someone before you thought about this issue and fixed it either in hardware or firmware or software, right? So, and yeah, in this session, we're focusing a bit on how we can use like smart IT systems to make sure we can overcome these transient failures. So when you are... creating or writing and building distributed applications you can run into all these issues so that's why about five and a half years ago some people at microsoft got together and created this open source project dapper the distributed application runtime and a lot of organizations use this to speed up their back-end development so typically like microservice development event driven architectures um and it's not only about about building it but also running it right so dapper typically runs on top of kubernetes so um yeah you it's not only beneficial for actually building your system you're coding it but it's also beneficial when you actually are running it and using it It came out of Microsoft Research, like I mentioned, like a couple of years ago, and the ownership was then transferred to the CNCF, the Cloud Native Computing Foundation. And since November last year, it's now a graduated project within the CNCF. And that's quite a big thing, because there's not a lot of graduated projects within the CNCF. So it's really a proven technology, because you need to go, you need to pass some really, some detailed audits, and they do like end user interviews for companies who are using Dapper. So it's really, really a proven technology. And yeah, very much still under active development as well and growing, growing every year. You see some logos here. I typically tell people to go to the CNCF website and go to client case studies, and you can then filter them on the project. So you can filter on all kinds of Dapper projects there. I think there were about 12 case studies described on the CNCF. So if you want to know, okay, is Dapper a real thing? Are the companies using it? Yes. Just look at the CNCF website. Yes. Alternative two? Is it an alternative to containerization? No, not really. I mean, so you are running a Dapper in a container with your application because you run it on top of Kubernetes. So it's more like working together. So it's not alternative, but hey, you indeed, you containerize your applications and then you can use it with Dapper. And I'm actually gonna, the next slide actually will make it a bit clearer, hopefully for you. No, no, good question, good timing as well. So we actually discussed it up front, so perfect. So. You can basically use Dapper with any language. So, of course, we're talking about .NET now and I will show .NET samples. But you can basically use Dapper with any language because Dapper is not really part of your application. Dapper runs next to your application in a separate process in a sidecar. And that's why it's typically run on Kubernetes because it's quite a familiar pattern in the Kubernetes world. um so yeah and as long as you can talk to the sidecar which which runs in in in the same in the same pot as long as you can communicate over http or gpc any language will will do dapper has a couple of client sdks to make it the communication a bit easier so there is like a.net sdk for dapper but if you don't want to introduce new sdks in your application you're fine just just have to make like pure http calls to dapper sidecar so that's that's possible So Dapper actually decouples your application from the underlying infrastructure. So Dapper offers a lot of APIs that does all of these white squares here. So for instance, if I take the PubSop API, so there's a PubSop API that you can use in your application, but Dapper itself is not a message broker, but you can configure Dapper to use like any message broker out there. So the benefit is that in your client code, you don't have like a direct dependency of a specific message broker. So you don't have any Azure Service Bus code in your application, but instead you rely on the Dapper API for PubSub messaging, which makes it really easy to switch the underlying. Matches Broker. So when you do local development, you can do an in-memory Matches Broker, which of course doesn't make sense when you're running a production, but it's great for local development. But when you move to production, then you would switch one configuration file, it's called the component file, and then you point to something that's running in Azure. So you become really flexible when you start using Dapper. And there's also lots of cross-counter concerns built in. So Debra has like built-in observability based on open telemetry. And there's also security built in. You can do like scoped access because this service can send messages to the message broker and this service can subscribe. These services are allowed to talk to each other, et cetera. And there's also resiliency built in. And if there's some time left, I will cover that as well. But I think we'll focus on this one on workflow. And like I mentioned, Debra is usually run on top of kubernetes and you can use any flavor of kubernetes out there right it doesn't really matter which cloud you use um have for azure there's also azure container apps so dapper is also partially compatible with container apps and there are a couple of versions behind so if you are running container apps i think it's 113 or something dapper is used there but currently dapper open source is at 115 and 116 is coming in like three or four weeks So if you want to use the latest and the greatest, then run it on top of Kubernetes. If you're okay with running a couple of versions behind, then you can also use it on Azure Container Apps. All right. So who's completely new to Dapper here? Okay, quite some folks. Okay, well, then this is my shameless plug. Part of my job is to create educational content about Dapper. For the last couple of months, I've created this Dapper 101 course, which sounds ideal for most of you. I will, again, share some links later, but this sounds ideal. Basically, we run a sandbox environment in the cloud, so you don't have to install anything. You just need to do a signup form, and then you get this VM in the cloud. You can run Dapper commands and stuff like that. You can explore a bit what's possible with Dapper. pepper all right end of the plug um getting back to our problem that we're solving right so we know systems are failing and ideally we need to like recover from failure but ideally automatically and limit the impact of the failure right so and that's where the durable execution concept comes in so durable execution guarantees that your code always runs to completion even if the process that runs the code fails and it sounds like very counterintuitive but What happens is like another process will be spun up and the code will actually continue running until it actually runs to completion. And that is possible because the state of the execution will be persisted to a state store. So, and maybe that sounds like familiar because maybe you're familiar with workflow systems. Did you ever use like a workflow tool? Yeah. Okay. Which one have you used? Oh yeah, yeah, yeah. Okay, yeah. Venus Rutherford Foundation was used here. Yeah, right. So personally, I've used like Azure Dribble functions a lot, which is actually like very similar to Dapper. And anyone here used Dribble functions? No? Okay, okay. so workflow systems they consist of three parts like an authoring environment to actually create the workflow and that can be either visual or in code there's actual workflow engine to actually schedule execute the workflow and there's like a state store to persist the workflow state so you need these three components And here I'm showing like an animation to see what kind of input and output is going on between the workflow engine and the state store. And so when you start a workflow, that workflow will be scheduled and the workflow has an ID. You also have an input and that's stored. And then the activity will be scheduled and executed and also that will be stored. But also when it's completed, then the output of the activity will also be stored. Then the workflow will not continue to activity two, but the workflow will replay from top. so it will first check okay is activity one already completed yes it will read that output from the state store back into the workflow then it will continue to the next one and again once it has been completed it will replay from top again and so on and so on and so here you see there's a lot of i o going on between the workflow engine and the stage door but also that your workflow uh will be actually re-running replaying a couple of times and that's very important to uh to know and because i will be working with a workflow as code solution and if you ever do some debugging of a workflow as code solution you put like a breakpoint on top you'll see that breakpoint will be hit like many more times than you would have expected and that's because the workflow is actually replaying and it's not like being run once from from a to z all right So now specific for Dapper workflow. So Dapper workflow also implements this Drupal execution principle. You can combine it with other Dapper APIs, which there are a lot of. I think we have like 12 different Dapper APIs now. And you can author workflows in actually all of the Dapper languages. So it's C Sharp and Java, Python, JavaScript, and Go. I'll be running some samples in C Sharp here. And you can apply all kinds of workflow-specific patterns, and I'll be covering them very soon. So here's a typical small workflow that you can create. It's usually really intended for automating business-like processes. Here an example here, your workflow will start because you will receive a message and then the first activity uses the Dapper API to retrieve a secret and then to make a service application call to another service. When you get back the response, then we actually are waiting until another event comes back into the workflow. so it's not about scheduling tasks but you can also do some some other stuff like waiting for incoming events and using timers and stuff like that and you can also run tasks in parallel as you can see here here we like storing an event and invoking another api and then the workflow will do a final activity and then publishing a message to another topic which maybe like kicks off something else yeah so this is like a very very typical but very small example of a workflow So now Debra authoring, Debra uses the WordPress code approach. So you actually write your workflow definitions in code in C-sharp. Again, that has pros and cons. I mean, personally, I like it. That's also the reason why I like Azure Drupal functions so much because I just can just write my workflows in code. It's part of Git, so it's version controlled. I can write unit tests for it. Yeah, that's what I like. But there are also some downside, right? So there's no immediate visual representation of the workflow. so so that can put some people off sometimes so maybe some people are familiar with logic apps in azure maybe yeah so it's like very visual um but yeah so that can be also nice to create but then then it's like very hard to to test and put it under source control so and there are other like workflows code solutions right so maybe some people are familiar with temporal that's quite a big player i think they're bigger in the java world than the net world though and i also already mentioned jibble functions as well if you're interested in these durable execution platforms or workflow engines in general this is a link to like an awesome list on github it lists like like dozens and dozens of these of these systems i don't recommend like trying them all out because you'll be busy for years probably okay so what kind of applications can you write with workflows well then we get into the to the workflow patterns because some sometimes you actually need to change your way of thinking or how you actually write your code so sometimes it requires a bit of a special approach approach and so the most basic form of writing workflow is to like task chaining and so this is really like i first want to do a and then b and then c and task chaining you use when the order of execution is important and that's probably because maybe here the output of a is using input for for b and so on so that's why the order is important there's also the fan out fan in pattern and here there's no dependency whatsoever between the activities so here we are not not chaining and not using the input and output for the next one but these can be run completely independently and these will be run as much as possible in in parallel the nice thing about workflows is that you can actually have the workflow actually wait until all of these activities have been completed and then you can still do some aggregation over the end result so maybe you do maybe do some has some different retrievals of different systems and then you can have some extra code in the workflow that does some analysis on all of those retrievals and the monitor pattern that's very useful when you do like reoccurring activities so maybe you want to run a nightly cleanup job in the cloud to get rid of some cloud resources so then you can use this this pattern so first you call an activity to which calls like another api to clean up your cloud resources and then there's a method on the on the workflow uh to create a timer and which can be set to anything that you want and once the timer is expired and there's another command that you can use that's called continue is new and that is and that instructs the workflow to continue as a new fresh instance so this has nothing to do with the replay functionality because the replay is still happening but continue as new actually instructs the workflow to start over as a really fresh instance you can forget about all of your historical executions and so this is more like a do while loop we can because you can also insert some if else logic in this as well Then there's the external system interaction one. This is particularly useful when you want to do some kind of approval step or have some kind of, so maybe you want to order like a fancy new laptop, but you are above a certain budget, so you need like manager approval. So then you can have an activity first to notify your manager, and then your manager uses like another system, and that actually sends an event back into this workflow instance, and this workflow will be waiting until that event comes in. and then you can inspect the payload of that event and based on that you can choose different branches of your workflow. This is the final pattern and it's child workflow. So what you can do is, of course, you can create like a huge workflow of like 100 different activities, but it gets like, yeah, I would say not very easy to watch and to look at. So what you can do is you can actually group activities that blow together or that you want to treat as reusable, and you can create those in child workflows. So in this case, I actually made a chaining pattern, but the chain consists of individual workflows. So yeah, this is just a nice way of actually grouping things together and make it more manageable to work with bigger workflows. Okay, a bit about the DEPR workflow engine. So the DEPR workflow engine is part of the DEPR sidecar, right? So the workflow engine is not part of your own application, but the workflow engine runs next to your application. And as soon as your application starts up and the DEPR sidecar starts up, a GRPC stream will be established between your application and the sidecar. And then the sidecar will check if your application has any workflow specific code. And if so, it will load that into the sidecar and it will check if there are any unfinished workflows. So once this is established, it will check the state store if there is any unfinished business. If so, then the Dapper sidecar will actually instruct the workflow to actually continue the work until it completes. So there's a lot of need, as I mentioned before, a lot of communication going on between your workflow app, between the workflow engine, but also between the workflow engine and the state store. Diving a little bit deeper into the Dapper sidecar. So the workflow engine is actually built on top of the Dapper Actors system. So the Dapper Actors is actually one of the other APIs of Dapper. And it's just... good to know that it has been developed like this because if you inspect all of the the logs you'll see a lot of mentions of of actors in the logs so don't be surprised about that and you're also not not supposed to directly communicate with those actors right because they're they're sort of internal although you can access them it's more internal usage so there is one actor type for orchestrators that will actually persist or the the workflow state to to the state store and there is one activity actor and so for each individual activity you'll have one instance of an actor of this activity type and that's also persisting its own state and so the workflow engine is actually communicating with two types of actors and two persisted states and to monitor what needs to be scheduled and executed just yeah just a nice to know so yeah it explains why you see actors in the logs okay almost time for the demo so um the first demo i'm showing it has like two applications we have a workflow application and there's a shipping service next to it and the workflow application is also interacting with an estate store in the inventory and this is the workflow that we're running so it's i called it the validate order workflow because it does a sort of like order validation and something with registering a shipment so when it starts we will inspect the payload and we will use an activity to check if you have enough inventory and actually already mute data inventory so this is the update inventory activity If there is sufficient inventory, then we continue and we will get a list of shipping providers by calling another activity. So we get back a list of shipping providers and then for each result of that shipping providers, we will get the cost for this shipment. So at the end of this, this will be a fan out, fan in situation. So at the end, when we have these three costs, we will select the cheapest shipper and we will actually register our shipments with this cheapest shipper. if this is successful then great we are at the end if this is not successful then I'm doing what it's called like a compensation action and we actually want to undo my inventory mutation so this so here I've mutated my inventory but if everything goes wrong and I can't register the shipments then I'm making a business decision and in this case the business decision is undo this update inventory So this is totally up to you, but the whole principle of compensation actions is you can apply compensation actions when something goes wrong in your workflow. But it's of course really a business decision what you actually do as a compensation action, right? So as a developer yourself, this needs to be agreed upon with the business, how actually of course this workflow looks like and what do you do when things are failing. I mean, an alternative could be I don't use the cheaper shipper because that phone probably fails, but I choose the next one of the who is the cheapest and try that and maybe it can be a loop. So I try everything until I have a successful one. So that's totally up to you. Okay, looking at the code. So this is my, I have a web application based on .NET 9. I only have one dependency and that's dapper workflow, the .NET package, because when you actually are. writing authoring workflows then you do need to use these Dapper client libraries right because yeah you're actually writing them in code so we cannot write them with just http commands all right so here's an example of the of a workflow so when you create a workflow it's just a plain.net class you're inheriting from a base type called workflow and it has like two generic arguments the input type and output type And when you implement this workflow, you actually need to override this run async method that's provided by the base type. And so you get this workflow context. So this is from Dapper. And this has all kinds of methods to call activities, to create timers, to continuous new, et cetera, et cetera. And you specify your input and the output is a task of whatever output you want it to be. And it's also realized that everything is async when it comes to workflows right because yeah a workflow might be very short and it might be ready in a second but it could also be a very long running workflow and it'll be ready within only in three months or so right so everything is asynchronous and yeah you cannot assume that it's like executed very very quickly and so here we are using this context to call another activity and we first then specify what kind of output type we expect back from that activity and then we specify the name of the activity class So in this case, it's the update activity that we're calling and we specify the input for that activity. So we get back a result and then we just have some if statements. Is there sufficient stock? If so, get me the list of shipping providers. So now you see I have one additional argument. So again, this is my return type. This is the activity that I'm calling. I'm creating a new object as my request for this activity. But now I also have this part. This is the workflow task option. So this is one of the additional features you can use with Dapper workflow is you can add retry policies. So in case this activity fails, you can actually instruct the workflow. keep on retrying it for as long as you want right you have different ways of doing it it can be like a constant retry it can be like an exponential retry that's completely configurable so this is my array of shipper information so that's the output of this what happens then is is part of this fan out fan in so what i first create is i create a list of tasks of the type of the activity that i'll be calling next and those are my shipping cost results So I have a list there. Then I'm iterating all of my different shipping providers. I'm creating a new request for that activity. And here I am calling that activity. But notice there is like no await here. So we're actually not actually scheduling and executing this task. We're just preparing this task and we're adding those individual tasks to this shipping cost result task list. So at the end of this for each, we have this. task which is now filled with a couple of providers and here here's where the magic happens here where we do await task when all this is where dapper engine actually schedules and actually executes all of these tasks and the workflow actually waits with with continuation until all of these results have come back oh yeah yeah yeah exactly comparable yeah exactly yeah indeed yeah so at this moment we have all the results and here i'm just simply doing a min buy and providing the cost to get me the whatever cheapest shipping service out there Then the final part is wrapped in a try catch, and this is to illustrate how you can deal with these compensation actions. So the next activity that I'm calling is the register shipment. So I'm registering the shipments because this is the cheapest shipping service. And if this fails, then this sketch block will be triggered. So the actual exception will actually be wrapped in a workflow task failed exception. So you have to catch the right type of exception that's important of course you can always inspect of course the inner exception etc and then i am calling this undo update inventory right and yeah there's no there's no magic happening here this is just another plane activity that i've written myself to undo the operation that i've done earlier All right. So this is the workflow itself. Let me just give an example of how to write an activity, which is like very similar. Again, it's a .NET class. You inherit from a workflow activity. You specify the input and output again. In this case, I am actually using the Dapper client because I'm interacting with the state management API of Dapper. So that gets injected here. you have to implement or override the run async method again to hamburg so this is the code that's actually being run when i'm calling this activity so here i'm using the depra client to actually get the state for a specific product id and when the product inventory actually comes back because maybe yeah it can it cannot be found so that's also an option but when the product inventory is not nil so it is found then i'm checking if the quality is enough to be ordered And if it is in order, I'm mutating that state and I'm using the Dapper API again to save that state. And since most of you are not familiar with Dapper, so I'm using a state store here, but from the code, you cannot see which state store it is, right? So I have a state store component name, but that's named inventory. So that doesn't give away anything. And the API is getStateAsync with the type that I'm expecting. And here is the safeStateAsync. So yeah, there's nothing visible in code that determines where is this like Cosmos DB or is this stable storage or whatever. We don't know yet because Dapper is actually abstracting that away for you. Okay, another example is the getShippingCostActivity. So here in this activity, we're actually calling out to another web service that I've created. So here we are injecting an HTTP client and this HTTP client is configured in such a way that it always sets a specific header and that's actually a named header so it actually knows which service I am communicating to. I'll show that a bit later. I'm simply doing a post to a calculate cost endpoint on the shipping service and providing this request and I get back a response which I'm just passing back to the workflow. So this is the program CS file of the shipping app. So this is the endpoint I'm actually calling. Nothing really weird is happening here. I'm intentionally doing like a threat sleep because I want to demonstrate something and I need some time to react. So hopefully your code will not contain this ever. And then I am creating like a new shipping result and I'm using some randomization to actually give me a nice new random value each time. okay then the final bit so this is so now we're back in the workflow application so this is our program cs file of our workflow application and so this is the setup for the http client that is used to communicate with our shipping service so whenever we are injecting the http client we are using this dapper method dapper clients create invoke http client and we are setting the application id and we set it to shipping so in in dapper terms all your developer applications always have an application id and that needs to be unique so i have a workflow application and that's the name is workflow and they have a shipping application and that id is shipping and by by by setting this up whenever we use the hp clients we are always targeting the shipping application And Dapper will actually figure out where this thing lives. So you don't have to deal with ports or with addresses. Dapper will figure it out for you. The final important thing of configuring this is there is this extension method, add Dapper workflow. You need to add this to your services collection because you need to instruct Dapper that this application contains workflows and activities. so if you don't do this dapper is unaware that you actually have a workflow application some other language is the case you don't need to do this but in.net you still have to do this maybe it will evolve over time so this setup is not needed anymore but now you still need to register your workflows and activities and at the moment your workflow and all of the activities that they need to be need to be part of one application right so you cannot have a workflow with activities like span across like different applications so that's not possible at the moment that will change though but what you typically do inside an application had there you make an http call to another service or in an application in an activity there you publish a message to another service and then you can do a pubsob back to the workflow application and because you can instruct the workflow and to wait for an incoming event so that's the way how you actually do cross service communication with workflows so this is the end point that i will be calling in a few seconds validate order we are passing it an order object then we are creating a scope and we are retrieving the dapper workflow client and we are using the method called schedule new workflow async So this is the moment where actually we say to the Debra workflow, okay, we have a workflow here with the name of ValidateOrderWorkflow. We are giving it an instance ID. So this is optional. If you don't give an instance ID, Debra will generate one for you. But if you have like another logical instance ID that you want to pass, in this case, I already have an order ID. So I'm reusing this order ID as a unique identifier for this workflow instance ID. And this is important because you need this instance ID when you want to check the status or you want to get rid of the old data and stuff like that. So this instance ID is very important. You need to capture that. And you're providing the input, which is the order. So what you get back, so you get back the instance ID, because everything is async, so you don't get back the final result of this workflow, but you get back the instance ID. In this case, I'm already providing it, but I'm capturing it back anyway, and I get back and accept that with the instance ID back to this endpoint. Now coming back to the state, so Debra works with component files, and in the component files, there is a definition of the underlying state that you're using. so in this case there's a component file and it's using state.redis as you can see here so when you install dapper locally you use the dapper cli to get everything up and running but you also get a couple of docker containers installed and one of them is a redis container and that's the default state store but also the default pubsob message broker when you run stuff locally can you use the Oh, I don't know the name, but the question is, can we use like the cloud native alternative to Redis? So yeah, DAPR is compatible with like dozens of different state stores. So I don't know the actual name of it, but yeah, so that's definitely described in the DAPR docs. There's a lot of state stores that DAPR is compatible with. and so it doesn't need to be this this specific one in this one in this case i'm just specifying like a local container it can also be something in the cloud or it can also be like a cloud where this server or whatever it can also be something completely different than redis so you have like literally dozens and dozens of options there yeah that definitely is yeah that definitely is yeah Yeah, definitely. Yeah, that's really also the reason why Dapper is designed. Even though it came out of Microsoft, it's really designed as like an open source cloud agnostic, language agnostic tool. So it can be used by anyone and everywhere. So yeah, but good question. All right, so now we know that Redis is used, which is running locally. And this, what you see here is another YAML file because if you're dealing with Dapper, you have to deal with YAML files, which is a bit unfortunate, but okay. and so this is a multi-app run file and i'm using this file to start my two applications and also make sure that there's a dapper sidecar to these applications so it has a bit of a common setup to make sure that i point to the path where this stavestoryyml is located i do some configuration of my application logs and my dapperd logs where i can find them and then there's an array of my applications in this case i have a workflow applications in this folder and I have a shipping application in this folder and I'm using head.net run to start both of these services. Alright, so what I'm doing now is I want to. simulate that there's like a very big issue which takes down all of my applications and dapper processes so what i'm going to run is i'm going to use this dapper multi-app run to start both my workflow app and my services app including dapper sidecars and i will make a request to this validate order endpoints the workflow will start but then i will immediately stop everything i just do ctrl c to to stop the whole dapper process which actually also stops my applications and to simulate like a serious issue then i will call dapper run again so my both my applications will start up again and then we will see what happens because then yeah you will see that actually the code will be executed until completion okay so i'm building my shipping app here and my workflow app and as you can see here so this is the the automation magic of demo time i'm just clicking a button here but in the background some some some terminal commands are running so that's the the power of that vs code extension demo time and basically my whole demo is like scripted which quite quite nice i can't make any typos anymore and so i'm now doing the dapper run f to actually start my workflow application and my shipping application so now everything is up and running there's a lot of output here of course that's configurable and the first thing i'm gonna do is i'm actually i didn't show you but there were also endpoints to make sure i have some inventory to actually make sure that this order validation actually succeeds so i first make sure that i am have enough inventory i can also check if i have enough inventory yeah i've got like five items of this rubber duck 1000 so that's all good and then i'm going to start this workflow um and again if you're not familiar with this i'm using another vs code extension called the the rest client okay so i see some people nodding that they know this but you can create these dot http or dot rest files and you can write your your requests in here and again you can also generate some random numbers and stuff like that so there's actually like a small button here called send request that actually makes my request to my application and i see the output here on the right so again i just love vs code So I'm going to click this. The workflow will actually start, but I also will stop it quite soon after that. Okay, making the request and now I'm stopping. Okay, so I'm going to scroll up a bit because we did see quite some logging, but the last logs were from our shipping application, which means that the workflow was communicating to our shipping application. And we see the logs that has shipping service B getting cost, service A getting cost. The workflow actually did the fan out part. It has gone into the shipping app, but now I've just stopped everything. So the workflow is down, the shipping app is down, and also no Dapper process up and running anymore. So now I will restart our applications, but I won't do any HTTP invoke stuff anymore, right? Because the whole state is now captured in Redis of the workflow state, which is not completed. So let's restart and let's see what happens. Okay, so things are starting up again. So now we're at the same point as we were previously. communicating with the shipping servers which had like a five second delay and now we see you have registered with shipping c and we already see in the logs that have completed so we see that the validation workflow completed with a completed status so we can already see it in the logs but we can also actually check the workflow status by calling to a specific endpoint in our depper sidecar so we're now calling out directly to the sidecar and providing the instance id of our workflow so if we do this then we get back the actual status of the workflow and we see that it's completed and we can see the entire input and the entire output all right so that works that's the power of dual execution right so everything was completely stopped but we can still start up the process and everything will just yeah it will run to completion like auto magically just because everything is stored in the state and had the workflow engine is able to figure it out So that's pretty powerful. For the next demo, I will change the code slightly to simulate an issue with this register shipment endpoint in order to trigger this other action. So what I will do is I will go to the program series file of our shipping application. And now it's just hard coded to always say, okay, this is fine. But we're actually going to change this to actually return a 500. So I need to build the shipping app again. change the code and I will run all of the applications again and then I need to restock my inventory again and I will start the workflow again but now I won't stop it I'll just run it to completion but then I'll inspect the logs and I will show you where you can see that actually the other activity was run to compensate for this action okay so we're making the requests we're getting all of the shipping costs here so that's this as usual we are using shipping a uh to do it uh and we we still see that um we have our workflow completed with orchestration status completed uh but it didn't complete in the in the regular way right because something went wrong and then we actually yeah yeah this is the task field successfully yeah exactly yeah exactly yeah yeah i should have i should have inserted this here it's like an like an oscure art in the log or something yeah i'll do this i'll do it next time and but here we see this undo update inventory activity yeah so here we can see that this activity to actually compensate the action was actually was actually triggered all right um okay stopping the application cool um so it has to this it's about like yeah designing or offering workflows and how they look like and running them but yeah running them is just one part you also need to need to think about workflow management that's quite often like an overlooked thing so dapper does have some workflow management capabilities for instance yeah all these we already did the starting one and getting the status one but you can do more as you can also pause a workflow and maybe that could be useful when you're in like a very long running workflow and maybe something later is not working if we need to pause this workflow before we get into problems later and that could be a reason to to pause a workflow you can also resume a workflow which is useful if you can pause it and you can also terminate a workflow, which means I'm actually sort of hard stopping the further execution of a workflow. And that's probably because something went wrong in a workflow and you want to reinitialize it yourself or rerun it yourself completely. And the final one is purging the state of the workflow. So terminating the workflow just terminates the execution of the workflow, but it doesn't remove all of its state from the state store. But purging the workflow is actually removing the state from the state store. and that's actually an important one because dapper doesn't do any purging of itself so if you just run workflows over and over again you will collect a lot of data over time and which does not get get purged unless you do something of course on your state store system to run a store procedure or whatever to clean up this data but if you're purely relying on dapper dapper doesn't do anything automatic for you so at the moment you are really you're really reliant on running this disperse command yourself and it's important to realize so these are the http endpoints to do all these things and this is also baked in into the individual language hdks so there's also a .NET API to do the same thing but for this example you don't need the .NET API you can actually make these raw http calls to the Debra sidecar from the application But here you see why this instance ID is so important that you actually get when you actually start a workflow. Because for all of the operations, you need a specific instance ID of this workflow. And at the moment, since you can only purge per instance ID, I definitely recommend to always store this. instance id somewhere else so you can actually keep track of what is already executed because there's no like batch cleanup you cannot say to the workflow engine okay you just clean up all of the completed jobs from last month or last year so that that operation is not available in in the in the workflow engine yet so i hope it will be there in the future so now you have to be really careful by capturing all this information it probably really depends again on like business requirements or service level requirements but what kind of state or data do you need to store still for your customers or stuff like that yeah so probably there's some agreement around that and you have to you have to make sure that you still keep this data for some time yeah yeah exactly exactly yeah yeah yeah and if that's not clear then definitely make it clear make it explicit uh yeah because yeah i i wouldn't go on and just do something yourself even if there's nothing on paper so i would definitely get some someone to validate this this process yeah well yeah it's a thing that that's often overlooked right because it's very nice and easy to create something but yeah once it runs in production yeah then then you're stuck with it right so it's better to think of these things up front yeah exactly yeah yeah you're also collecting lots of data over time yeah okay uh there are some more workflow uh challenges and i'm gonna give you some tips for it so this is not specific to dapper workflow but more general in workflow systems and typically for workflow systems as code in general workflow should be deterministic meaning for every time you start a workflow with the same input it should result in the same output And this has to do with what I showed earlier in this animation, because the workflow will be replaying many, many times. And so all of the code in your workflow will be executed many, many times. So if that somehow gets into like a different state, because you have state from your state store, but also we need state during runtime. So if there's a mismatch, that will be guaranteed, it will give you some issues in your workflow state. So make sure that you use deterministic code in your workflow. If you have non-deterministic code, always wrap it inside an activity. Because anything in an activity can be anything. And that's because all of the inputs and the outputs of an activity are persistent to disk anyway. So as soon as it's been run once the activity, it's stored in the state store. And then after the replay, the data will be just rehydrated from the state store. I'll just give you an example. So this is an example of a non-deterministic workflow. So for instance, using like GUID new GUID directly in a workflow is a no-go, right? Because as soon as this hits like the second time, you'll have like a new order ID compared with the first execution, right? So don't do this. Luckily, there's like a helper method on the workflow context. So that's called context new GUID. So there's like a replacement method that you can use. And the other no-go is using stuff like daytime UTC now. So now timestamps, of course, this will also always be different. Luckily there's like the current UTC daytime property on the context as well. So that's also good. It is good to know that in the .NET SDK, this is there. but for instance there are different languages the case and they are not all at the same level yet so for instance java and python i think they do have the currency to see day daytime but they don't have to go with method for instance so if you're using that workflow in a different language sometimes you are you're forced to either make a poor quest and of course help help us maintain dapper or wrap it still in an activity okay so the question how to deal with duplicate data um yeah that has to do with one of the other tips i'll be sharing more like idempotency instead of deterministic data so i'll get to it in a minute yeah yeah So yeah, for some things it's built into the workflow context. So that's great. For other stuff, just wrap it in an activity and then you're done. Okay, so now coming back to your question. So all the code in activities should be idempotent, meaning if you run it multiple times, you should not result in any side effects. And one of the side effects could be like, how do you deal with like... had the same data being generated for instance well yeah it's really up to you to make sure that that doesn't happen you know so for instance when you want to if you're like inserting a record in a database maybe just don't blindly do it but maybe first do a read before you do a write for instance or maybe you can do like an upsert instead of an insert so no matter how many times it will be run yeah it will just be an absurd instead of an insert yeah exactly so but yeah but it is important to think about that and not only when it comes to like inserting records in a database but also calling out other apis right so are like third-party apis are your third-party apis idempotent is it okay to make a second request exactly the same to that third-party api so yeah it also might need some some investigation in your dependencies at the moment Yeah, one of the really the most tricky things is like workflow versioning when it comes to workflows as code. Because it's too easy to actually introduce a breaking change in your workflow. And when you introduce a breaking change, that means you have some workflows. Let's say I'm making a change to workflow and I'm deploying that new workflow to my application. But there are still workflows which are unfinished, which are called in-flight. So those unfinished workflows have stayed in a state store, but I've already deployed a new version. And that new version is like different models or different order of activities compared to what I have in my state store. So these two, they don't match anymore. So again, that leads to probably like runtime issues because I'm reading and try to match old data in my new workflow definition. I think you can give this talk. That's on my next slide. This is good. No. So I got a suggestion about what about doing canary deployments? Well, that's one of their three solutions. so let's start with your ones so one of them is definitely to do like canary deployments do like blue green deployments using different applications definitely the safest option i think um but i think it's also the most complex option right because and probably also the most expensive i don't know you have to like have more more more things more infrastructure to to spin up and stuff like that uh but but it's definitely good um but it also still requires you to look at uh had the old application to see okay when are all of these uh workflows when are they completed So yes, it works, but you still have to do some inspection to see if everything is completed. And at the moment, the Dapper Management API doesn't really provide a good way to report on that. So we have all kinds, we have like very specific instance ID based methods to get status, but you can't simply do like a query, give me a count of how many unfinished workflows there are. So that's not there yet. So it's also very hard to monitor that. Okay, so let's go from bottom to top now. So the other one is maybe you have the luxury to wait until there are no more in-flight workflows. I think that's not very realistic that it happens, but maybe you only have like a lot of work during the day and maybe during the night, nobody is using your system. That could be a solution, definitely not a very safe one, but then again, you still rely on monitoring the status of all of your workflows. So I think the most- most basic one is actually to include like a version suffix in your workflow version naming because I actually didn't show you that but I'll show you that now I'm going to go to my my redis visualizer and to show you how this state is stored yeah So redis, it's like key value based data, right? So what we see here are the keys of how the data is stored. So it has a prefix of the Dapper application ID, which is very boringly called workflow. Then there's like a double pipe. And then there's like a whole string, which is Dapper dot internal default workflow. And then again, the name of the workflow that's been started. So again, there's the app ID. And then there's the workflow instance ID. So it's a big concatenated string. But yeah, but this workflow thing, you could, right now it's just workflow, but ideally, of course, if you version your workflow, these versions will be, these names, these keys will be different, right? So in that case, it doesn't matter if your application then comes back because you still have your version one there, but you also have your version two there. So all of the in-flight ones can still use the version one code and all of the new requests that come in can use the version two. So that is the safest. There's of course, there's a little downside that there is of course a client that actually schedule starts the workflow that needs to be updated to also use this new name. And typically that client is part of the same application, but it could also be living in another application to start your workflow. So you have like two places that need to be aware of this new workflow name. So that's a potential downside. I don't think it's very big, but you do need control over that client that it has a new name. um yeah here's just an example so here's a version uh workflow one um and my input is a string my output is an int and i'm using this order id as the input for these two activities um and but in my second version i'm using the order item as an input for a result a but then i'm using result a as the input for result b yeah so this is this is already a good example of a breaking change of workflow okay final tip and then and then i'm done and so the final tip is to keep your payloads small because we already saw there's a lot of input output going on to the state store and so it really makes sense to keep your payloads for these workflows and activities as small as possible ideally just ids or very small pocos because there's a lot of like serialization, deserialization going on. And here I have an example of a large payload size workflow to update a very large document. So here in this first activity, I am doing a get document. So what I get back is maybe like a huge JSON document. So this is like stored as an output for this activity, but then I'm using this document as an input for the next activity. So this payload will be stored twice, once as an input and once as an output. So that's also not very, very efficient. So in those cases, I would say just wrap those two things into one activity, right? So just be, don't be very like, okay, an activity can only do like one method execution. No, just group some things inside an activity to not have very chatty workflows. All right, I'm going to stop the resiliency part because I've already been talking way too long. uh i'm going to share some qr codes if you want to capture the qr codes uh be be ready so i've already talked about this so uh deputy university one thing uh so at the moment there's a dapper 101 course i can definitely recommend that uh there's also a dapper workflow one uh and here and in this workflow one i think most of it's already covered in this one sorry yes yeah yeah yeah so yeah so had so this uses a online sandbox and you you run dapper in the sandbox So you're not writing code, but you're definitely like running all kinds of Debra commands, you're running applications, you're inspecting the logs. So it's really like a hands-on stuff. And there's also like a Debra agent's lesson at the moment. There's more if you're interested in AI stuff. I'm currently still working on the Debra and .aspire course with someone else. I think it should be live in like two or three weeks. And I'm also working on a running Debra in production course as well. All right, and some other stuff I have to share. I'll wait until you're done. No? You can just Google Debra University. Okay, okay. Another thing, if you're new to DEPA workflow and you have a hard time scaffolding all these applications codes, because we have to create these workflows and activities and stuff like that, it can be a bit meh. We have an online workflow composer, and then you can either upload a picture or you can use an online drawing tool, which is also backed in, and then click a button, and then it will actually create your .NET code or Java code or Python code to do the workflow scaffolding for you. So it'll generate like a complete workflow, which is runnable, but you still have to implement your own logic in the activity code, of course. Okay, so this is the one that actually points to the GitHub repo, which actually contains this. So this might be useful if you want to run this yourself. It also comes with a dev container configuration. So you can run it in GitHub code spaces if you want, or just locally. And you can also run the complete demo time stuff as well, like I've been doing now. And finally, well, you've been listening for me for quite a long time, so I apologize. If you want, you can claim the Dapper Community Supporter badge. They also appear there as a sticker form, but this is like a digital form. Actually, if you help to contribute to Dapper any form, either code contributions or like docs contributions, or if you give presentations about Dapper, we give away all these digital badges that you can claim. All right, I think it's time. Any other questions? Latent, if there are any latency and performance impacts. So there are metrics published on performance nature for every Debra release. So I think those are related to service to certification. So if you do service, you go from service to Debra sidecar to Debra sidecar to service. I think the average latency was about like five or six milliseconds. So it's not that bad. Yes.

Marc Duiker is coming to Liverpool to present a topic on durable execution + Dapr

Talk
Applications break all the time, there could be a network issue, a cloud provider outage, or just a glitch in the matrix. But as a developer, you really need your applications to be resilient without the need to recover databases and restart services manually.

In this session, I'll demonstrate how Dapr Workflow provides durable execution, which enables you to write reliable workflows as code. In addition, I'll show how resiliency policies in Dapr improve reliable communication across services and resources when developing distributed applications.

I'll go into specific workflow features, such as scheduling, sequential and parallel execution, and waiting for external events. I'll show many code samples (in C#) for each of these features and will run the applications using the Dapr CLI to demonstrate their resiliency.

By the end of the session, you will have a good grasp of how durable execution with Dapr workflow and resiliency policies can help you build resilient applications.