Failure is not an option: durable execution + Dapr = 🚀 with Marc Duiker

Yeah, so no clicker thing because I'm running everything from VS code or any VS code users here Yeah, so some VS code. Okay. Yeah, so yeah I really like VS code probably because I'm not like running like mega solutions in the full video I don't really maybe need it, but I like all the extensions of VS code So I'm actually using a demo time quite a new extension which allows you to actually create your whole slide deck in VS code, but it's also like an VS code automation tool because I'm actually running Dapper applications also via demo time. Really cool extension created by Microsoft MVP, Elios Drive. Yeah, pretty cool. But we're not here for that. We're here for this talk. Failure is not an option. Drupal execution plus Dapper is rocket emoji. It's amazing, I would say. So let's go. Maybe some of you are as old as I am, I have seen this information message in real life. Task failed successfully in Windows XP. I've definitely seen it, which is worth a very silly thing to display to your end users, right? I mean, why would you ever, ever show this? I'm actually not sure what triggered is this information message, but. i think it's pretty obvious that we know that failure is inevitable right it happens every day sometimes we don't know it is happening but definitely failure is happening and um yeah it is actually the topic of this talk right because yeah failure is happening but you want to limit the impact to the users of these failures right so either limit impact or maybe the user doesn't even notice that there was a failure so that's what the dual execution and dapper talk is about A little bit about me. I'm a developer advocate for a diagram. It's a US based startup and it's founded by Mark fossil and you won Snyder, the co-creators of Dapper open source. And I'm also one of the Dapper community managers. So we do like monthly live streams where we like invite some maintainers or contributors to highlight new features, or we invite Dapper end users to show their solution and how they are running it in production. And I'm also always looking for people who are actually running Dapper in production because I do like interviews with them and I also co-author like client case studies for the CNCF, the Cloud Native Computing Foundation. So if you are someone who is using Dapper in production or you know someone who is using Dapper in production, please point them to me. I'm also an Azure MVP. um for quite some years now and and i have a lot of like creative hobbies so i do a lot of pixel art so some of the stickers you find at the back of the table i've created them and i've also like created lots of profile pictures in pixel art form maybe you still see them somewhere on x or on some other social media if you want to see more go to mark duiker.dev But let's get back to the topic of today. And when I did some research on IT failures, I came across this blog post. It's called Lessons from a Decade of IT Failures. It's already quite an old blog post. It's from 10 years ago, 2015. um it's on a very respectable blog the ieee spectrum blog and i really highly recommend you all to to to visit this because it's like an interactive report on on the lessons and learnings of the decade of it failures from 2005 to 2015. i'm hoping going to make a new edition this year for for the last 10 years But I just took a screenshot of that blog post and I just want to highlight the impact of the monetary cost of all of the issues over these 10 years. So if you see here, you see a legend. So the biggest circle is worth 10 billion in US dollar cost. And here is then a chart over 10 years and then across different geographies. And then you see like various size circles, small from very big. So, and the total like monetary costs really runs in the dozens and dozens or probably even like hundreds of billions of dollars in cost. And that was like 10 years ago. So, um, maybe in the last 10 years, I don't think that ever decreased. I wouldn't be surprised if it's bigger now, but a lot of things are going wrong in it and I'm not claiming by following this session, you'll be able to solve all of them. Uh, and we're just gonna focus on, on one specific type of failures. Um, but I, I found this blog post for like, very intriguing to it to read. And one quote from this blog post said, um, while a rogue algorithm led the Knights capital group to lose 440 million in Aaron trades in about 45 minutes in 2012. so in just like three quarters of an hour you can lose like so many millions just by like a wrong um wrong code change exactly yeah so no definitely um yeah and by the way um yeah i'm running this all of these codes and this is backed by a git repository i'll share the link later so you don't have to take pictures i'll share some qr codes at the end so you can get to this git repo um right So, one of the reasons said that things are difficult is because a lot of us are making distributed applications and this is quite a simple 1. is like a typical ecommerce application with the head shopping cart inventory, checkouts, payment, shipping, etc. So, this is still relatively basic, but a lot of. Things can go wrong in these type of systems. You have a lot of moving parts, a lot of services need to communicate with each other. And what happens if one service is like temporary down, different types of storage, the message brokers, observability is also an issue typically. So this is really, yeah, this is part of the problem that also our systems are becoming so complex. And this is nothing new really because in 1994 two people, Peter Deutsch and James Gosling, created this fallacies of distributed computing. Did you ever hear about this fallacies? Yeah, some people are familiar with this. yeah and so it's it's a set of false assumptions programmers new to distributed applications often make yeah so all of these assumptions are of course like incorrect but people new to distributing yeah they they simply don't know or they don't think about these things and wow and this is not a theoretical session so i'm not going to cover all of these eight points uh i'm definitely going to highlight some of the links i've shared here here below especially the bottom one is like a youtube playlist from odin from particular software who really very clearly explains all of these fallacies so definitely have a look at that um it's also very important to know that not all failures are the same so in this case i am looking at technical issues and not more like operational or other type of issues but when it comes to technical issues you have like transient failures which are like are temporary and usually resolve themselves and you have like permanent failures and those are yeah indeed as the name says permanent they are not temporary they don't resolve themselves But when they are transient, meaning they are resolving themselves, of course, there's nothing. There's no magic involved there. They are transient because someone before you thought about this issue and fixed it either in hardware or firmware or software. Right. So and we are in this session, we're focusing a bit on how we can use like smart IT systems to make sure we can overcome these transient failures. So when you are. creating or writing and building distributed applications you can run into all these issues so that's why about five and a half years ago some people at microsoft got together and created this open source project dapper the distributed application runtime and a lot of organizations use this to speed up their back-end development so typically like microservice development event driven architectures um and it's not only about about building it but also running it right so dapper typically runs on top of kubernetes so um yeah you it's not only beneficial for actually building your system you're coding it but it's also beneficial when you actually are running it and using it It came out of Microsoft Research, like I mentioned, like a couple of years ago, and the ownership was then transferred to the CNCF, the Cloud Native Computing Foundation. And since November last year, it's now a graduated project within the CNCF. And that's quite a big thing, because there's not a lot of graduated projects within the CNCF. So it's really a proven technology, because you need to go, you need to pass some really, some detailed audits, and they do like end user interviews for companies who are using DAPR. So it's really, really a proven technology. And yeah, very much still under active development as well and growing, growing every year. You see some logos here. I typically tell people to go to the CNCF website and go to client case studies and you can then filter them on the project. So you can filter on all kinds of Dapper projects there. I think there were about 12 case studies described on the CNCF. So if you want to know, okay, is Dapper a real thing? Are the companies using it? Yes. Just look at the CNCF website. Yes. Alternative two. Is it an alternative to containerization? No, not really. I mean, so you are running a Dapper in a container with your application because you run it on top of Kubernetes. So it's more like working together. So it's not an alternative, but you indeed, you containerize your applications and then you can use it with Dapper. And I'm actually going to, the next slide actually will make it a bit clearer, hopefully for you. No, no, it's a good question. Good timing as well. So we actually discussed it up front. So perfect. So. You can basically use Dapper with any language. So of course, we're talking about that now and I will show the samples, but you can basically use Dapper with any language because Dapper is not really part of your application. Dapper wants next to your application in a separate process in a sidecar. And that's why it's typically run on Kubernetes because it's quite a familiar pattern in the Kubernetes world. So yeah, and as long as you can talk to the sidecar, which runs in the same pod, as long as you can communicate over HTTP or GPC, any language will do. Dapper has a couple of client SDKs to make communication a bit easier. So there is like a .NET SDK for Dapper. But if you don't want to introduce new SDKs in your application, you're fine. You just have to make like pure HTTP calls to Dapper sidecar. So that's possible. So Dapper actually decouples your application from the underlying infrastructure. So Dapper offers a lot of APIs that does all of these white squares here. So for instance, if I take the Pub-Sub API, so there's a Pub-Sub API that you can use in your application, but Dapper itself is not a message broker, but you can configure Dapper to use like any message broker out there. So the benefit is that in your client code, you don't have like a direct dependency of a specific message broker. So you don't have any Azure service bus code in your application, but instead you rely on the Dapper API for PubSub messaging, which makes it really easy to switch the underlying. Matches broker. So when you do like local development, you can do like an in memory message broker, which of course doesn't make sense when you're running a production, but it's great for local development. But when you move to production, then you would like switch one configuration file. It's called a component file and then you point to something that's running in Azure. So you become really flexible when you start using using Dapper and there's also lots of. Cross-counter concerns built in so Deborah has like built in observability based on open telemetry and there's also security built in. You can do like scoped access because I had this service can again send messages to the message broker and this service is can subscribe. These services are allowed to talk to each other, etc. And there's also resiliency built in and if there's some time left, I will cover it as well. But I think we'll focus on this one on workflow. And like I mentioned, Deborah is usually run on top of Kubernetes and you can use any. flavor of Kubernetes out there right it doesn't really matter which cloud you use um have for Azure there's also Azure container apps so dapper is also partially compatible with with container apps and there are a couple of versions behind so if you are running container apps I think it's 1.13 or something dapper is used there but currently dapper open source is at 1.15 and 1.16 is coming in like three or four weeks So if you want to use the latest and the greatest, then run it on top of Kubernetes. If you're okay with running a couple of versions behind, then you can also use it on Azure Container Apps. All right. So who's completely new to Dapper here? Okay, quite some folks. Okay, well, then this is my shameless plug. Part of my job is to create like educational content about Dapper. So for the last couple of months, I've created like this Dapper 101 course, which sounds like ideal for most of you. So I will again share some links later, but this sounds ideal. Basically, we run a sandbox environment in the cloud, so you don't have to install anything. You just need to do like a signup form and then you get this VM in the cloud. You can run Dapper commands and stuff like that. So you can explore a bit what's possible with that. All right, end of the plug, um, getting back to our problem solving, right? So we know systems are failing and ideally we need to like recover from failure, but ideally automatically, uh, and limit the impact of the failure. Right? So, and that's where the, uh, durable execution, uh, concept comes in. So, durable execution guarantees that your code always wants to completion. Even if the process that runs the code fails, and it sounds like very counterintuitive, but. What happens is like another process will be spun up and had the code for actually continue running until it actually wants to completion. And that is possible because the state of the execution will be persisted to a state store. So, and maybe that sounds like familiar because maybe you're familiar with workflow systems. Did you ever use like a workflow tool? Yeah. Okay. Which one of you use? Oh yeah yeah yeah okay yeah. The foundation was was used here yeah right so personally I've used like Azure Dribble functions a lot which is actually like very similar to to Debra and anyone here used Dribble functions. No okay. So workflow systems, they consist of three parts, like an offering environment to actually create the workflow and that can be either visual or in code. There's an actual workflow engine to actually schedule, execute the workflow and there's like a space store to persist the workflow state. So you need these three components. And here I'm showing like an animation to see what kind of input and output is going on between the workflow engine and the state store. So when you start a workflow, that workflow will be scheduled and the workflow has an ID. We also have an input in that storage and then the activity will be scheduled and executed and also that will be stored. But also when it's completed, then the output of the activity will also be stored. Then the workflow will not continue to activity two, but the workflow will replay from top. So it will first check, okay, is activity one already completed? Yes, it will read that output from the state store back into the workflow. Then it will continue to the next one. And again, once it has been completed, it will replay from top again and so on and so on. And so, hey, you see there's a lot of I.O. going on between the workflow engine and the state store, but also that your workflow will be actually rerunning, replaying a couple of times. And that's very important to know because we'll be working with a workflow as code solution. and if you ever do some debugging of a workflow as code solution you put like a breakpoint on top you'll see that breakpoint will be hit like many more times than you would have expected and that's because the workflow is actually replaying and it's not like being run once from from a to z all right so now specific for for for debra workflow has of debra workflow also implements this turbo execution uh principle and you can combine it with other dapper apis and which are a lot of i think we have like 12 different dapper apis now And you can offer workflows in like all of you and actually all of the Dapper languages. So it's C sharp and Java, Python, JavaScript and go. I'll be running some samples in C sharp here and you can apply all kinds of your workflow specific patterns and I'll be covering them very soon. So and so here's like a typical small workflow that you can create. It's usually really intended for automating like business like processes, right? So here an example here, your workflow will start because you will receive a message and then the first activity uses the Dapper API to retrieve a secret and then to make a service application call to another service. And when you get back the response, then we actually are waiting until another event comes back into the workflow. So it's not about scheduling tasks, but you can also do some other stuff like waiting for incoming events and using timers and stuff like that. And you can also run task in parallel as you can see here, here we like storing an event and invoke another API and then the workflow will do a final activity and then publishing a message to another topic, which maybe like kicks off something else. Yeah, so this is like a very, very typical, but very small example of a workflow. So now Debra authoring Debra use the WordPress code approach. So you actually write your workflow definitions in code in C sharp. Again, that has pros and cons. I mean, personally, I like it. That's also the reason why I like Azure Drupal functions so much because I can just write my workflows in code. It's part of Git, so it's version controlled. I can write unit tests for it. Yeah, that's what I like. But there are also some downside, right? So there's no immediate visual representation of the workflow. So that can put some people off sometimes. So maybe some people are familiar with logic apps in Azure maybe. So it's like very visual. Um, but yeah, so that can be also nice to create, but then then it's like very hard to to test and put it under the source control. So, and there are other like workflows code solutions, right? So maybe some people are familiar with temporal. That's quite a big player. I think they're bigger in the Java world than the network though. And I also already mentioned jibble functions as well. If you're interested in this job execution platform for workflow engines in general, this is a link to like an awesome list on GitHub. It lists like dozens and dozens of these systems. I don't recommend trying them all out because you'll be busy for years probably. Okay, so what kind of applications can you write with workflows? Well, then we get into the workflow. patterns and because some sometimes you actually need to change your way of thinking or how you actually write your code so sometimes it requires a bit of a special approach approach and so the most basic form of writing workflow is to like task chaining and so this is really like i first want to do a and then b and then c and task chaining you use when the order of execution is important and that's probably because maybe here the output of a is using input for for b and so on so that's why the order is important There's also the fan out fan in pattern and here, there's no dependency whatsoever between the activities. So here we are not not chaining and not using the input and output for the next one, but these can be run completely independently and these will be run as much as possible in parallel. The nice thing about workflows is that you can actually had the workflow actually wait until all of these activities have been completed and then you can still do some aggregation over the end result. So maybe you do. Maybe do some of the has some, some different, um, heavy retrievals of different systems and then you can have some extra code in a workflow that does some analysis on all of those retrievals. Um, the monitor pattern that's very useful when you do like a reoccurring activities. So maybe you want to run a nightly cleanup job in the cloud to get rid of some cloud resources. Um, so then you can use this, this pattern. So, first you call an activity to which calls like another API to clean up your cloud resources. Uh, and then there's a method. on the on the workflow to create a timer and which can be set to anything that you want and once the timer is expired and there's another command that you can use that's called continue as new and that is and that instructs the workflow to continue as a new fresh instance so this has nothing to do with the replay functionality because the replay is still happening, but continue as new actually instructs the workflow to start over as a really fresh instance. You can forget about all of your historical executions. And so this is more like a do while loop. We can, because you can also insert some if else logic as well. Then there's the external system interaction one. This is particularly useful when you want to do some kind of approval step or have some kind of, so maybe you want to order like a fancy new laptop, but you are above a certain budget, so you need like manager approval. So then you can have an activity first to notify your manager, and then your manager uses like another system, and that actually sends an event back into this workflow instance. And this workflow will be waiting until that event comes in. and then you can inspect the payload of that event and based on that you can choose different branches of your workflow. This is the final pattern and it's child workflow. So what you can do is, of course, you can create like a huge workflow of like 100 different activities, but it gets like, yeah, I would say not very easy to watch and to look at. So what you can do is you can actually group activities that blow together or that you want to treat as reusable, and you can create those in child workflows. So in this case, I actually made a chaining pattern, but the chain consists of individual workflows. um so yeah this is just a nice way of actually grouping things together and make it more manageable to work with bigger workflows okay a bit about the depra workflow engine um so the depra workflow engine is part of the depra sidecar right so it's a little bit workflow engine is not part of your own application but the workflow engine runs next to your application and as soon as your application starts up and and your the depra sidecar starts up a grpc stream will be established between your application and the sidecar And then the sidecar will check if your application has any workflow specific goals. And if so, it will load that into the sidecar and it will check if there are any unfinished workflows. So once you've once this is established, it will check the stage. So if there is any unfinished business, if so, then the sidecar will actually instruct the workflow to actually continue the work until it completes. And so there's a lot of need, as I mentioned before, a lot of communication going on between your workflow app, between the workflow engine, but also between the workflow engine and the state store. Diving a little bit deeper into the Dapper sidecar. So the workflow engine is actually built on top of the Dapper actor system. So the Dapper actors is actually one of the other APIs of Dapper. And it's just. Good to know that it has been developed like this, because if you inspect all of the logs, you'll see a lot of mentions of of actors in the logs. So don't be surprised about that. You're also not not supposed to directly communicate with those actors because they're sort of internal, although you can access them. It's more internal usage. So there is one actor type for orchestrators that will actually persist the workflow state to the state store and there is one. activity actor and so for each individual activity you'll have one instance of an actor of this activity type and that's also persisting its own state so the workflow engine is actually communicating with two types of actors and two persisted states and to monitor what needs to be scheduled and executed just yeah just a nice to know so yeah it explains why you see actors in the logs okay almost time for the demo so um the first demo i'm showing it has like two applications we have a workflow application and there's a shipping service next to it And the workflow application is also interacting with an estate store in the inventory. And this is the workflow that we're running. So it's I called it the validate order workflow because it does a sort of like order validation and something with registering a shipment. So when it starts, we will inspect the payload and we will use an activity to check if you have enough inventory and actually already mute data inventory. So this is the update inventory activity. If there is sufficient inventory, then we continue and we will get a list of shipping providers by calling another activity. So we get back a list of shipping providers and then for each result of that shipping providers, we will get the cost for this shipment. So at the end of this, this will be a fan out, fan in situation. So at the end, when we have these three costs, we will select the cheapest shipper and we will actually register our shipments with this cheapest shipper. If this is successful, then great, we are at the end. If this is not successful, then I'm doing what it's called like a compensation action and we actually want to undo my inventory mutation. So here I've mutated my inventory, but if everything goes wrong and I can't register the shipments, then I'm making a business decision. And in this case, the business decision is undo this update inventory. So this is totally up to you, but the whole principle of compensation actions is you can apply compensation actions when something goes wrong in your workflow. But it's of course really a business decision what you actually do as a compensation action, right? So as a developer yourself, this needs to be agreed upon with the business, how actually of course this workflow looks like and what do you do when things are failing. I mean, an alternative could be I don't use the cheaper shipper because that phone probably fails, but I choose the next 1 of the cheapest and try that and then maybe it can be a loop. So I try everything until I have a successful 1. so that's totally up to you. Okay, looking at the code. So this is my application based on 9. I only have 1 dependency and that's that step or workflow. That don't net package because when you actually are. writing authoring workflows then you do need to use these uh depper client libraries right because yeah you're actually writing them in code so we cannot write them with just http commands all right so here's an example of the uh of a workflow so when you create a workflow it's just a plain uh dotnet class uh you're inheriting from a base type called workflow and it has like two generic arguments the input type and output type And when you implement this, this, this roughly you actually need to override this one async method that's provided by the base type. And so you get this workflow context. So this is from the, from, from, from wrapper. And this has all kinds of methods to call activities, to create timers, to continuous new, et cetera, et cetera. And you specify your input and the output is a task of whatever output you want it to be. And it's, yeah, it's also realized that everything is async when it comes to. workflows right because yeah a workflow might be very short and it might be ready in a second but it could also be a very long running workflow and it'd be ready within only in three months or so right so everything is asynchronous and yeah you cannot assume that it's like executed very very quickly and so here we are using this context to call another activity and we first then specify what kind of output type we expect back from that activity and then we specify the name of the activity class So in this case, it's the update activity that we're calling and we specify the input for that activity. So we get back a result and then we just have some if statements. Okay, is there sufficient stock? If so, okay, get me the list of shipping providers. So now you see I have like one additional argument. So again, this is my return type. This is the activity that I'm calling. I'm creating like a new object as my request for this activity. But now I've also have this part. This is the workflow task option. So this is one of the additional features you can use with Dapper workflow is you can add like retry policies. So in case this activity fails, you can actually instruct the workflow. keep on retrying it for as long as you want, right? You have different ways of doing it. It can be like a constant retry. It can be like an exponential retry. That's completely configurable. So this is my array of shipper information. So that's the output of this. What happens then is part of this fan out, fan in. So what I first create is I create a list of tasks of the type of the activity that I'll be calling next. And those are my shipping costs results. So I have a list there. Then I'm iterating all over my different shipping providers. I'm creating a new request for that activity. And here I am calling that activity. But notice there is like no await here. So we're actually not actually scheduling and executing this task. We're just preparing this task and we're adding those individual tasks to this shipping cost result task list. So at the end of this for each, we have this. Task, which is now filled with a couple of providers and here here's where the magic happens here where we do a wait task when all this is where Dapper engine actually schedules and actually execute all of these tasks and the workflow actually waits with with continuation until all of these results have come back. Oh, yeah, yeah, yeah, exactly. Yeah, yeah. So at this moment we have all the results and here I'm just simply doing a min by and provide the cost to get me to have whatever cheapest shipping service out there. Then the final part is wrapped in a try catch and this is to illustrate how you can deal with these compensation actions. So the next activity that I'm calling is the register shipment. So I'm registering the shipments because this is the cheapest shipping service. And if this fails, then this sketch block will be triggered. So the actual exception will actually be wrapped in a workflow task failed exception. So you have to catch the right type of exception. That's important. Of course, you can always inspect, of course, the inner exception, etc. And then I am calling this undo update inventory, right? And there's no magic happening here. This is just another plain activity that I've written myself to undo the operation that I've done earlier. All right. So this is the workflow itself. Let me just give an example of how to write an activity, which is like very similar. Again, it's a .NET class. You inherit from a workflow activity. You specify the input and output again. In this case, I am actually using the Dapper client because I'm interacting with the state management API of Dapper. So that gets injected here. we have to implement or override the run async method again to hamlet. So this is the code that's actually being run when I'm calling this activity. So here I'm using the Dapper client to actually get the state for a specific product ID. And when the product inventory actually comes back, because maybe it can be found, so that's also an option. But when the product inventory is not nil, so it is found, then I'm checking if the quantity is enough to be ordered. And if it is in order, I'm mutating that state and I'm using the Dapper API again to save that state. And since most of you are not familiar with Dapper, so I'm using a state store here, but from the code, you cannot see which state store it is, right? So I have a state store component name, but that's named inventory. So that doesn't give away anything. And the API is a get state async with the type that I'm expecting. And here is the. safe state async so yeah there's nothing visible in code that determines where is this like costless db or is this stable storage or whatever um yeah we don't know yet because dapper is actually abstracting that away for you okay another example is the get shipping cost activity so here in this activity we're actually calling out to another uh web service that i've created and so here we are injecting an http Clients and this HDP client is configured is in such a way that it always sets a specific header and that's actually a named header. So it actually knows which service I am communicating to. I'll show that a bit later. Um, I'm simply doing a post to a calculate cost endpoint on the shipping service and providing this request and get back a response. Um, which I'm just passing back to the workflow. So this is the program CS file of the shipping app. So this is the endpoint I'm actually calling. Nothing really weird is happening here. I'm intentionally doing like a thread sleep because I want to demonstrate something and I need some time to react. So hopefully your code will not contain this ever. And then I am creating like a new shipping result and I'm using some randomization to actually give me a nice new random value each time. Okay, then the final bit. So now we're back in the workflow application. So this is our program.cs file of our workflow application. So this is the setup for the... HTTP client that is used to communicate with our shipping service. So whenever we are injecting the HTTP client, we are using this dapper method, dapper client to create invokeHTP client, and we are setting the application ID and we set it to shipping. So in dapper terms, all your Debra applications always have an application ID that needs to be unique so I have a workflow application and that's the name is workflow and they have a shipping application and that ID is shipping and by by by setting this up whenever we use the HP clients we are always targeting the shipping application and Debra will actually figure out where this thing lives so you don't have to deal with with boards or with with addresses, Dapper will figure it out for you. And the final important thing of configuring this is there is this extension method, add Dapper workflow. You need to add this to your services collection because you need to instruct Dapper that this application contains workflows and activities. So if you don't do this, Dapper is unaware that you actually have a workflow application. Some other languages, the case you don't need to do this, but in .NET, you still have to do this. it will evolve over time so this setup is not needed anymore but now you still need to register your workflows and activities and at the moment your workflow and all of the activities that they need to be need to be part of one application right so you cannot have a workflow with activities like span across like different applications so that's not possible at the moment that will change them But what you typically do inside an application, there you make an HTTP call to another service or in an activity there you publish a message to another service. And then you can do a pop-up back to the workflow application. And because you can instruct the workflow to wait for an incoming event. So that's the way how you actually do cross-service communication with workflows. So this is the endpoints that I will be... calling in a few seconds validate order we are passing it an order object and then we are creating a scope and we are retrieving the dapper workflow client and we are using the method called schedule new workflow async so and this is the moment where actually have we say to the dapper workflow okay we have a workflow here with the name of validate order workflow we are giving it an instance id so this is optional if you don't give an instance id dapper will generate one for you But if you have like another logical intensity that you want to pass in this, in this case, I already have an order ID. So I'm reusing this order ID as a unique identifier for this workflow instance ID. And this is important because you need this instance ID when you want to check the status or you want to get rid of the old data and stuff like that. So this instance ID is very important. You need to need to capture that. And you providing the input, which is the order. um so what you get back and so you get back the intensity because everything is async so you don't get back the final result of this workflow but you get back the instance id in this case i'm already providing it but i'm capturing it back anyway and i get back in like an accepted with with the instance id back to this endpoint um now now coming back to the state so debor works with component files and in the component files there is the definition of the underlying state that you're using so in this case there's a component file and it's using state.redis as you can see here so when you install dapper locally you use the dapper cli to get everything up and running but you also get a couple of docker containers installed and one of them is a redis container and that's the default state store but also the default pubsob message broker when you run stuff locally can you use the Oh, I don't know the name, but the question is, can we use the cloud native alternative to Redis? So, yeah, Dapper is compatible with dozens of different state stores. So I don't know the actual name of it. But yeah, so it's definitely described in the Dapper docs. There's a lot of state stores that Dapper is compatible with. and so it does need to be this this specific one in this one in this case i'm just specifying like a local container it can also be something in the cloud or it can also be like a cloud reddish server or whatever it can also be something completely different than redis so you have like literally dozens and dozens of options there yeah that definitely is yeah that definitely is yeah Yeah, definitely. Yeah, that's really also the reason why Debra is designed, even though it came out of Microsoft is really designed as like an open source cloud agnostic language agnostic tool. So it can be used by anyone and everywhere. So yeah, but good question. All right. So now we know that Redis is used, which is running locally. And this, what you see here is another YAML file because if you're dealing with Debra, you have to deal with YAML files, which is a bit unfortunate, but okay. And so this is a multi app run file and I'm using this file to start my two applications and also make sure that there's a Dapper sidecar to these applications. So it has a bit of a common setup to make sure that I point to the path where this stavestory.yaml is located. I do some configuration of my application logs and my Dapper D logs where I can find them. And then there's an array of my applications. In this case, I have a workflow applications in this folder. Worldwide folder and I have a shipping application in this folder and I'm using .NET run to start both of these services. All right. So what I'm doing now is I want to. simulate that there is like a very big issue which takes down all of my applications and dapper processes so what i'm going to run is i'm going to use this dapper multi-app run to start both my workflow app and my services app including dapper sidecars and i will make a request to disvalidate order endpoints the workflow will start but then i will immediately stop everything i just do ctrl c to to stop the whole dapper process which actually also stops my applications and to simulate like a serious issue then i will Go Dapper run again. So my both my applications will start up again and then we'll see what happens because then you will see that actually the code will be executed until completion. Okay. So I'm building my shipping app here and my workflow app. And as you can see, so this is the automation magic of demo time. I'm just clicking a button here, but in the background, some, some, some terminal commands are running. So, so that's the, that's the power of that VS code extension demo time. And basically my whole demo is like scripted, which. quite quite nice i can't make any typos anymore and so i'm now i'm doing the debor run f to actually start my workflow application and my shipping application so now everything is up and running there's a lot of outputs here of course that that's configurable and the first thing i'm gonna do is i'm actually i didn't show you but there were also endpoints to make sure i have some inventory to actually make sure that this order validation actually succeeds so i'll first make sure that i am I have enough inventory. I can also check if I have enough inventory. Yeah, I've got like five items of this rubber duck 1000. So that's all good. And then I'm going to start this workflow. And again, if you're not familiar with this, I'm using another VS Code extension called the REST client. Okay, so I see some people nodding that they know this, but you can create these .htp or .rest files. You can write your requests in here. And again, you can also generate some random numbers and stuff like that. So there's actually like a small button here called send request that actually makes my request to my application. And I see the output here on the right. So again, I just love VS Code. so i'm going to click this um the workflow will actually start but i also will stop it quite soon after that um okay making the request and now i'm stopping okay so i'm going to scroll up a bit because we did see uh quite some logging but the last logs were from our shipping application which means that the workflow was communicating to our shipping application and we see the logs that has shipping service b getting cost service a getting cost so The workflow actually did the fan out part. It has gone into the shipping app, but now I've just stopped everything. So the workflow is down, the shipping app is down and also no Dapper process up and running anymore. So now I will restart our applications, but I won't do any HTTP invoke stuff anymore, right? Because the whole state is now captured in Redis of the workflow state, which is not completed. So let's restart and let's see what happens. Okay, so things are starting up again. So now we're at the same point as we were previously. communicating with the shipping servers which had like a five second delay and now we see you have registered to be with shipping c and we already see in the logs and have completed so we see that the validation workflow completed with a completed status so we can already see it in the logs but we can also actually check the workflow status by calling to a specific endpoint in our depra sidecar so we're now calling out directly to the sidecar and providing the instance id of our workflow so if we do this then we get back the actual status of the workflow and we see it's completed and we can see the entire input and the entire output all right so that works that has the power of dual execution right so everything was completely stopped but we can still start up the process and everything will just yeah it will run to completion like auto magically just because everything is stored in the state and had the workflow engine is able to figure it out so that's that's pretty powerful um for the next demo i will um change the code slightly to simulate an issue with this register shipment endpoint in order to trigger this um this this other action so what i will do is i will go to the program cs file of our shipping application and now it's just hard coded to always say okay this is fine but we're actually gonna gonna change this to actually return a 500. so i need to build the shipping app again because i've changed the code and i will run all of the applications again and then I need to restock my inventory again and I will start the workflow again but now I won't stop it I'll just run it to completion but then I'll inspect the logs I will show you where you can see that actually the other activity was run to compensate for this action okay so we're making the requests we're getting all of the shipping costs here so that's business as usual we are using shipping a uh to do it uh and we we still see that um we have our workflow completed with orchestration status completed uh but it didn't complete in the in the regular way right because something went wrong and then we actually yeah yeah this is the task field successfully yeah exactly yeah exactly yeah yeah i should have i should have inserted this here it's like i'm like an oscure art in the log or something yeah i'll do this i'll do it next time and but here we see this undo update inventory activity so here we can see that this activity to actually compensate the action was actually was actually triggered all right um okay it's stopping the application cool um so it has to this about like yeah designing or offering workflows and how they look like and running them but yeah running them is just one part you also need to need to think about workflow management that's quite often like an overlooked thing So Dapper does have some workflow management capabilities. For instance, all these, we already did the starting one and getting the status one, but you can do more as you can also pause a workflow. And then maybe that could be useful when you're in like a very long running workflow and maybe something later is not working. If we need to pause this workflow before we get into problems later. And that could be a reason to pause a workflow. You can also resume a workflow, which is useful if you can pause it. And you can also terminate a workflow, which means I'm actually sort of hard stopping the further execution of a workflow. And that's probably because something went wrong in a workflow and you want to reinitialize it yourself or rerun it yourself completely. And the final one is purging the state of the workflow. So terminating the workflow just terminates the execution of the workflow, but it doesn't remove all of its state from the state store. But purging the workflow is actually removing the state from the state store. And that's actually an important one because Dapper doesn't do any purging of itself. So if you just run workflows over and over again, you will collect a lot of data over time and which does not get purged unless you do something, of course, on your state store system to run a store procedure or whatever to clean up this data. If you're purely relying on Dapper, Dapper doesn't do anything automatic for you. So at the moment you are really, you really reliant on running this disperse command yourself. And it's important to realize, so these are the HTTP endpoints to do all these things. And this is also baked in into the individual language SDKs. So there's also a .NET API to do the same thing. But for this example, you don't need the .NET API. You can actually make these raw HTTP calls to the Dapper Sidecar from the application. But here you see why this instance ID is so important that you actually get when you actually start a workflow, because for all of the operations, you need a specific instance ID of this workflow. So, and yeah, and at the moment, since you can only perch per instance ID, I definitely recommend to always store this. instance id somewhere else so you can actually keep track of what what is already executed and because there's no like batch cleanup you cannot say to the workflow engine okay you just clean up all of the completed jobs from last month or last year so that that operation is not available in in the in the workflow engine yet so i hope it will be there in the future so now you have to be really careful by capturing all this information It probably really depends again on like business requirements or service level requirements, but what kind of state or data do you need to store still for your customers or stuff like that? So probably there's some agreement around that and you have to make sure that you still keep this data for some time. Yeah, exactly. Yeah, yeah, yeah. And if that's not clear, then definitely make it clear, make it explicit. uh yeah because yeah i i wouldn't go on and just do something yourself even if there's nothing on paper so i would definitely get some someone to validate this this process yeah yeah well yeah it's a thing that that's often overlooked right because it's very nice and easy to create something but yeah once it runs in production yeah then then you're stuck with it right so it's better to think of these things up front yeah Yeah, exactly. Yeah. You're also collecting lots of data over time. Yeah. Okay. There are some more workflow challenges and I'm going to give you some tips for it. So this is not specific to Dapper workflow, but more general in workflow systems and typically for workflow systems as code. In general, workflow should be deterministic, meaning for every time you start a workflow with the same input, it should result in the same output. And this has to do with what I showed earlier in this animation, because the workflow will be replaying many, many times. And so all of the code in your workflow will be executed many, many times. So if that somehow gets into like a different state, because you have state from your state store, but also we need state during runtime. So if there's a mismatch, that will be guaranteed, it will give you some issues in your workflow state. So make sure that that is you use like deterministic code in your workflow. If you have non deterministic code, always wrap it inside an activity. Yeah, because yeah, anything in activity can be anything and that's because all of the inputs and the outputs of an activity are persistent to disk anyway. So as soon as it been run once the activity it's stored in the state store and then after the replay and the data that we just rehydrated from the state store. Just to give you an example, so this is an example of a non-deterministic workflow. So for instance, a use like GUID new GUID directly in a workflow is a no-go, right? Because as soon as this hits like the second time, you'll have like a new order ID compared with the first execution, right? So don't do this. Luckily, there's like a helper method on the workflow context. So that's called context new GUID. So there's like a replacement method that you can use. and the other no-go is using stuff like daytime UTC now so now timestamps of course this will also always be different luckily there's like the current UTC daytime property on the context as well so that's also good it's it's it is good to know that in the in the net SDK this is there but for instance there are different language SDKs and they are not all at the same level yet so for instance Java and Python I think they do have the current UTC daytime but they don't have the GUID method for instance so if you're using that workflow in a different language sometimes you are you are forced to either make a poor quest and of course help help us maintain dapper or rapid still in an activity okay so the question how how to deal with duplicate data um yeah that has to do with one of the other tips i'll i'll be sharing more like uh idempotency uh instead of uh deterministic data so i'll get to it in a minute yeah yeah um and so yeah for some things it's built into the workflow context so that's great for other stuff just wrap it in activity and then you're done okay so now coming back to your question so all the code and activities should be idempotent meaning if you run it multiple times you should not result in any side effects and one of the side effects could be like how do you deal with like Uh, the same data being generated, for instance, um, well, yeah, it's really up to you to make sure that that doesn't happen. Uh, you know, so for instance, when you want to, if you're like inserting a record in a database, uh, maybe just don't blindly do it, but maybe first do a read before you do a right. for instance or maybe you can do like an upsert instead of an insert so no matter how many times it will be run yeah it will just be an absurd instead of an insert yeah exactly so but yeah but it is important to think about that and not only when it comes to like inserting records in a database but also calling out other apis right so are like third-party apis are your third-party apis idempotent is it okay to to make a second request exactly the same to the third-party api so yeah it also might need some some investigation in your dependencies at the moment Yeah, one of the really the most tricky things is like workflow versioning when it comes to workflows as code. Because it's too easy to actually introduce a breaking change in your workflow. And when you like introduce a breaking change, that means you have some workflows. Let's say I'm making a change to workflow and I'm deploying that new workflow to my application, but there are still workflows which are unfinished, which are called in flight. So those unfinished workflows have stayed in a state store, but I've already deployed a new version and that new version is like different models or different order of activities compared to what I have in my state store. So these two, they don't match anymore. Right, so again, that leads to like probably like runtime issues because I'm like reading and try to match like all data in my new workflow definition. I think you can give this talk. That's on my next slide. This is good. No. So I got a suggestion about what about doing canary deployments? Well, that's one of their three solutions. So let's start with your ones. So one of them is definitely to do like canary deployments, do like blue green deployments using different applications. Definitely the safest option, I think. But I think it's also the most complex option, right? Because probably also the most expensive. I don't know, you have to like have more things, more infrastructure to spin up and stuff like that. But it's definitely good. But it also still requires you to look at the old application to see, okay, when are all of these workflows, when are they completed? So, so, so yes, it works, but you still have to do some inspection to see if everything is completed. And at the moment, the Dapper management API doesn't really provide a good way to, to report on that. Yeah. So we have all kinds, we have like very specific instance ID based methods to, to get status, but you can't simply do like a query of, give me, give me a count of how many unfinished workflows there are. So that's not, that's not there yet. So it's also very, very hard to monitor that. Okay, so let's go from bottom to top now. So the other one is maybe you have the luxury to wait until there are no more in flight workflows. I think that's not very realistic that it happens, but maybe you only have like a lot of work during the day and maybe during the night. Nobody is using your system. That could be a solution. Definitely not a very safe one, but then again, you still rely on him monitoring the status of all of your workflows. So I think the most. The most basic one is actually to include like a version suffix in your workflow version naming. Because I actually didn't show you that, but I'll show you that now. I'm going to go to my Redis visualizer to show you how this state is stored. Yeah. So Redis, it's like key value based data, right? So what we see here is the keys of how the data is stored. So it has a prefix of the Dapper application ID, which is very... Yeah, very boringly called workflow. Then there's like a double pipe. And then there's like a whole string, which is dapper.internal default workflow. And then again, the name of the workflow that's been started. So again, there's the app ID. And then there's the workflow instance ID. So it's a big concatenated string. But yeah, but this workflow thing, you could, right now it's just workflow. But ideally, of course, if you version your workflow, these versions will be. these names, these keys will be different, right? So in that case, it doesn't matter if your application then comes back because you still have your version one there, but you also have your version two there. So all of the in-flight ones can still use the version one code and all of the new requests that come in can use the version two. So that is the safest. There's of course, there's a little downside that there is of course a client that actually schedules, starts the workflow that needs to be updated to also use this new name. and typically that client is part of the same application but it could also be living in another application to start your workflow so you have like two places that need to be aware of this new workflow name so that's a potential downside i don't think it's very big but you do need control over that client that it has a new name um yeah here's just an example so here's a version workflow one um and my input is a string my output is an int and i'm using this order id as the input for these two activities. But in my second version, I'm using the order item as an input for result A, but then I'm using result A as the input for result B. So this is already a good example of a breaking change of workflow. Okay, final tip, and then I'm done. And so the final tip is to keep your payloads small. because we saw there's a lot of input output going on to the state store and so it really makes sense to keep your payloads for these workflows and activities as small as possible ideally just ids or very small pocos Um, because there's a lot of like serialization, deserialization going on. And, um, here I have an example of, um, a large payload size workflow, uh, to update a very large documents. So here in this first activity, I am doing a get documents. So what I get back as well, maybe like a huge Jason document. So this is like stored as an output for this activity, but then I'm using this document as an input for the next activity. So this, this payload to be stored twice, once as an input and once as an output. And so that's also, and not very, very efficient. So, in those cases, I would say just to just wrap those 2 things into 1 activity, right? So just be don't be very like, okay, an activity can only do like 1 method execution. No, just group some things inside an activity to not have very chatty workflows. All right, I'm going to stop the resiliency part because I've already been talking way too long. uh i'm going to share some qr codes if you want to capture the qr codes uh be be ready so i've already talked about this so uh deputy university one thing uh so at the moment there is a dapper 101 course i can definitely recommend that uh there's also a dapper workflow one as in here and in this workflow one i think most of it's already covered in this one sorry yes yeah yeah yeah so yeah so had so this uses a online sandbox and you you run dapper in the sandbox So you're not writing code, but you're definitely like running all kinds of Dapr commands, you're running applications, you're inspecting the logs. So it's really like a hands-on stuff. And there's also like a Dapr agent's lesson at the moment. It's more if you're interested in AI stuff. I'm currently still working on the Dapr and .aspire course with someone else. I think it should be live in like two or three weeks. And I'm also working on a running Dapr in production course as well. All right. And some other stuff I have to share. I'll wait until you're done. No? You can just Google Debra University. Oh, okay, okay, okay. Another thing, if you're new to DEPA workflow and you have a hard time scaffolding all these applications codes, because we have to create these workflows and activities and stuff like that, it can be a bit like meh. We have an online workflow composer, and then you can either upload a picture or you can use an online drawing tool, which is also backed in, and then click a button, and then it will actually create your .NET code or Java code or Python code to do the workflow scaffolding for you. So it'll generate like a complete workflow, which is runnable, but you still have to implement your own logic in the activity code, of course. Okay, so this is the one that actually points to the GitHub repo, which actually contains this. So this might be useful if you want to run this yourself. It also comes with a dev container configuration. So you can run it in GitHub code spaces if you want, or just locally. And you can also run the complete demo time stuff as well, like I've been doing now. And finally, well, you've been listening for me for quite a long time, so I apologize. If you want, you can claim the Dapper Community Supporter badge. They also appear there as a sticker form, but this is like a digital form. Actually, if you help to contribute to Dapper any form, either code contributions or like docs contributions, or if you give presentations about Dapper, we give away all these digital badges that you can claim. All right, I think it's time. Any other questions? Latent, if there are any latency and performance impacts. So there are metrics published on performance nature for every DEPRA release. So I think those are related to service to certification. So if you do service, you go from service to DEPRA sidecar to DEPRA sidecar to service. I think the average latency was about like five or six milliseconds. So it's not that bad. Yes.

Applications break all the time, there could be a network issue, a cloud provider outage, or just a glitch in the matrix. But as a developer, you really need your applications to be resilient without the need to recover databases and restart services manually.

In this session, I'll demonstrate how Dapr Workflow provides durable execution, which enables you to write reliable workflows as code. In addition, I'll show how resiliency policies in Dapr improve reliable communication across services and resources when developing distributed applications.

I'll go into specific workflow features, such as scheduling, sequential and parallel execution, and waiting for external events. I'll show many code samples (in C#) for each of these features and will run the applications using the Dapr CLI to demonstrate their resiliency.

By the end of the session, you will have a good grasp of how durable execution with Dapr workflow and resiliency policies can help you build resilient applications.

Learn Dapr with Dapr University: https://www.diagrid.io/dapr-university

The GitHub repo with samples and slides (includes a devcontainer and DemoTime instructions) : https://github.com/diagrid-labs/dapr-resiliency-and-durable-execution