Durable Execution in DevOps: How Uniphar Built Reliable Syst... Alice Gibbons & Vaclav (Oisin) Haken

Thank you so much for coming. Today we are going to talk to you about durable execution in DevOps. Specifically, I have my friend from Unifar here and we are going to talk about how they built distributed systems that were reliable with Dapper. And I have a clicker. And I'm going to use my clicker. Oh, I did it twice. Okay. Thanks so much for coming. My name is Alice Gibbons. I'm head of the customer engineering team at Diagrid, which is the company behind the open source Dapper project. And yeah, we're going to talk a lot about reliability in distributed systems today, specifically around workflow. And with me, I have my friend Oshin. Hello, my name is Oshin, and I work for Unifor, and I'm a member of a cloud developer. I work for kind of DevOps and platform engineering. so yeah we're going to show you how we use dupper right so i probably should start a little bit with you know just telling you what unifar actually is so unifar was really uh like a small little kind of cop of few pharmacists back in 1967 and it kind of grew a little bit you know so now it's going to be a big anniversary coming soon next year and it's it's now in 160 plus countries and 200 plus multinational partners and the whole idea is really just to you know give people critical life-saving drugs when they need to and just as well do it in just the easiest way possible which ultimately you know translates to cost and so really cover the whole end-to-end give you a lot of expertise that we've accumulated so far tailor solutions to your needs and ultimately do it now in a modern way because Originally it would have been Tony calling John and ordering stuff over the phone and now it's all going to be or is already digital. Awesome. And yeah, if you haven't heard of Diagrid before, as I said, we're the company behind the open source Dapper project, which is now a graduated project in the CNCF. And essentially our primary goal is to build platform and services for distributed and reliable applications, including AI agents, workflows, everything, that from there. We are, as I said, part of the CNCF as well as the AAIF. And yeah, we want to talk a lot about reliability today. Okay. this particular example solution is a lot to do with cost management and that's how we refer to it and ultimately you know when I described Unifor, I mentioned that it kind of grew over all those decades. And a lot of it was by acquiring other companies. But as you probably know yourself, acquiring can be a tricky business, especially because the IT systems that you acquire might have completely different tech stack and are in a different stage of evolution. So ultimately, now with... things moving into cloud and being modernized a whole new initiative was started multi-year project to move things into cloud securely modernize it and to enable all these kind of digital digital future streams and so the cost management is really an important aspect because it would be very easy to just migrate some beefy servers from on-prem to cloud and you know eventually though you get the bill right because clouds can be very expensive it's very easy to just clicky clicky or you know automate provisioning of resources but the bill then comes and you may be surprised so there's a big push onto okay there has to be value delivered for the value that you spend on it so obviously management wants to keep track of it and accounting people just as well so there were obviously some solutions built into azure We found that we wanted to go a little bit deeper, especially because some of that infrastructure may be shared between various different streams, so we wanted to allow for some kind of splitting of a cost between these initiatives. But at the end of the day, you get your budget for the project or for the stream, and the accounting people then see how much you've spent that previous month, and they can track whether you're actually, you know, above or under the budget. So that's really what this is about. So in terms of Azure, you would get your hierarchy starting from your tenant to your resource group, to your subscriptions, and then obviously at the bottom of it, the actual resources. And when we realized very quickly that as more and more resources appear in the marketplace or we start using them, we wanted something that can be fully extended to support all these new types. just the number of resource groups will grow, a number of new projects starting and their resources being provisioned. So we wanted something that would allow us to be highly distributed because ultimately for us to split some of that cost, it means calling some APIs with obviously all the usual. problems of potential timeouts, rate limits, or just your basic transient failures. And we also wanted to write that code in the sense that allow us to quickly react to new things, new strategies, new resources. So you have to kind of divide and conquer the solution instead of having some sort of spaghetti call with some intertwining streams of code. And the workflows were kind of the solution here. because they allow us to just run things under control, run whatever needs to happen inside, can be paralyzed. So ultimately for us it made sense to sort of just you know fan out all these discoveries, all the data discovery, data crunching and eventually fan in the numbers and at the end of the day you get your your report with the number attached to customer which is something that business understands accounting people understand and at the end of the day even the the implementators of the solution understand so normally how this works is that we have a kind of a monthly cron schedule right because ultimately you want on the first of the next month you want to crunch the numbers for the previous month and so that's that's kind of the request. Then we came up with a concept of a cost center rule file. The idea was that all these projects are started, they evolve, and it shouldn't really be one person or one team that controls that. We wanted even business people to be able to say, okay, I have these resources that I'm talking to my guys in the team. They're going to provision certain stuff. This is how we're going to deploy it. and ultimately I can introduce all these new rules and to make sure that that cost is projected to my budget. So that's kind of one of the inputs and when that all process finishes you get a CSV file because we found that to be very agnostic to underlying software for accounting processes. Yeah, and I think just to call it here, this is a common thing we see in terms of trying to bridge the gap between, like, the business and the technology that's being used at an organization. And I think this is a really good example of it. Like, you want a standard kind of CSV file that you can give to accounting in the language they understand, but then from the technology perspective, you can write your application in code. You can, like, live in the world that you currently live in and understand. And maybe if you can go back, sorry, I wanted to mention the example of that rule you see here uses the resource group that is, kind of most frequently used scenario we do support other levels you know even your basic tags or you know resource name full id match that sort of so so it's very flexible and so this is kind of a visual outline of of of this particular solution so this is where some of these features come into play so you see that at the top there's a tenant workflow which governs subscription workflow or workflows, right? That's the configuration of what you want to scan. And then you have a bunch of other downstreams, generally kind of your resource group one is where most of these kind of normal, you know, common resources. would be catered for. But we do want it also to provide some very special tailored logic. So we do have, for example, the specialized workflows for AKS splitting, where we just scan the pods and the CPU and memory metrics and kind of compute kind of a score. We do have VDI workflows, so based on the sessions and which groups you belong to, we can map you to, cost center. So we can easily extend it. That's the whole idea here. And just as long as that hierarchy of these kind of workflows is maintained, that works. Yeah, and I think this also lends itself well to, if you remember that kind of slide we showed before that shows the hierarchy of resources within Azure. So, you know, you start at your tenant level and then you come down. This, like... that is exactly mirrored by the type of applications you can build within workflow. So you can see that this kind of mirrors that, you know, tenant, subscription, resource group hierarchy in the same way as you would see in that, yeah, in the Azure space. Yeah. And you can see the green bar there to the left. So when these workflows, when they do their logic, they crunch the numbers, eventually the cost is split. The data that split is sent to a specialized actor which is you know and I sort of unit of isolation with specific ID scheme and eventually it allows us to to kind of control and because it's an the the actor contract gives you that kind of transactionality all these workflows can independently talk to it but but we do have control over properly appending the resources to the state that it maintains. So it allows us later to possibly look into that state for, you know, there are sometimes queries, you know, how come my bill went up so much last month? And we can then look at the underlying data and say, look, this resources provision, that's where the bulk of that comes from. Okay, so let's see a demo. Let's see a demo. Right. So first, I'm going to start the application itself. And when it started, you can also see my kind of a, so normally, like I said, the cron would run on the first of the next or every month. Obviously here, we're not going to wait that long. So I'm just going to trigger a workload kind of ad hoc. So you can see that you just sent some data that normally would be kind of defaulted for you based on that logic. But here I'm just going to, to generate it on my own. And now you can see the application is connected. So I can just trigger that process. And so Oshin's running his application locally here as well as using a kind of a remote dapper sidecar in this case. And we have the usual demo gremlins. Okay, let me just try again. But essentially, yeah, this could be run anywhere. We are just using an external Dapper sidecar to give you additional visibility into what is going to be the workflow app. But specifically, he's launching his application locally, and then we can see that in a dashboard here. And we did try this on the Wi-Fi before, so it was working. So I need to, it's probably... Probably some networking authentication. So let me just switch to our alternate strategy. So ultimately, this is kind of the view of what would happen. So the main workflow here is the tenant one, and there's some runs, and we ran some of it earlier. So this is kind of what would happen in terms of the flow. You can see that the tenant started. couple of children workflows or subscriptions and in them the other Sort of specialized workflows would happen. So if I try to just zoom in a little bit Just so you have an idea so you can see that for example special MySQL kind of logic or log analytics workspace logic Would be would be executed and eventually when it all kind of fans in then that goes in and sends the report. Yeah, so essentially this is like one parent workflow and then each of the child workflows that it kicks off is its own separate process. So you have a number of these sub-process workflows that are specific to each as a resource that you want to get your cost management and your stakeholder visibility into. And then the parent workflow is the one kicking that off and then running that. So that's what these child workflows are showcasing. Yeah, I can show you just some of the code that would kind of represent these building blocks. So when we set cron, triggers a request. This would be the controller. The controller simply does few checks. And at the end of the day, it uses the workflow SDK to schedule a new workflow. And that's the tenant one. If I look inside. then you can see that this one simply start a number of subscription workflows and just that again the level of isolation there and subscription workflow then runs a few things such as obviously you can obtain all these costs and then runs the resource group levels and all the specialized things. So you already saw kind of that MySQL one, the AKES one. So we can easily extend it in the future. And that's the structure right now and kind of what matches that catalyst field. Yeah, and I think the important thing to call out here in the code is like specifically he is just like using the adapter.net SDK and then wrapping his business logic within kind of like this durable execution wrapper, meaning that If we do kick it off and it does end up working here, we can stop and then restart that and it'll continue to run to completion. And so this, that's a long exception. But essentially what happens is you can write any of your business logic within the workflow definition and then within there, expand out and see each of the child workflows running. So try it one more time. See if that works one more time. And then, yeah. See if the Wi-Fi plays a little nicer with us. Live demos, you know? Okay, so the application's running. We're connected to our kind of remote sidecar here. That's what we're seeing. And then if we just try and kick one of these off, just with that curl command. Let's see. Now it's still, I think it might be authentication problem now. Oh, really? Okay. Okay. Let's go, okay, well, let's... That's too bad. I guess the other thing we wanted to talk about a little bit was just like what we kind of showcased in terms of the code. So if you haven't heard of Dapper before, it is an application runtime for building distributed systems. There is a number of APIs that support various design patterns in this space. So, you know, one of the things we're talking a lot about today is specifically for durable execution, which is specific to the workflow API. But there is a number of work other APIs that are also supported for things like message brokers, for things like PubSub, request for... request reply patterns, things like that. And specifically, it acts as an abstraction in between your infrastructure and your code. So if you have an application and you are depending on any sort of infrastructure resource, the dapper sidecar or the framework sits in between your app and your infrastructure there. So this means a ton of things. It means composability, modularity, it has cross-cutting concerns, things like resiliency, observability, and... security built in, but then not only that, you can also take advantage of as few or as many of these building blocks. So yeah, what Oceane is using within its application is specific to workflow, and this is kind of where a lot of these applications are going from a durable execution perspective. So essentially, when you kind of kick off one of these new apps, you can stop it, or if that process goes down for whatever reason, it'll pick back up from where it was running or where the last command it executed. So in terms of... Oh, is that it? I suspect that might easily be the problem. All right, we're doing some live debugging now. This is exciting. No, no, no, this is good. I want to see the demo for sure. You think you got logged out in 10 minutes. So essentially, like the... The main piece of the Durable Execution wrapper, though, is this checkpointing and replaying mechanism. So once, hopefully we get this to work here, but essentially the application as it's running is using the built-in Dapper workflow engine, which has a state store. And that state store is essentially an append-only log, meaning that as it's running, it's going to be checkpointing consistently its state to the state store. And then as per every single workflow activity, which is where all the business logic is being done. So in his case, it was... you know, where that kind of where we're hitting the Azure API, which apparently we're not logged into and pulling back the pulling back kind of the cost management data in that case. So any of those business logic runs, those are going to be activities and that's going to be stored into the state store as we go. And then when we pick back up, say they know that process goes down for whatever reason, it will continue to run to completion here. How are we doing? I think we might have a winner here. Let's have a quick look. And I'm actually kind of loving this because you never pick when your session's going to time out. So now we know. It was 20 minutes. Ah, perfect. So see, this is now the running one, right? Woo! Running! Thank you. I appreciate it. Now, what I'm going to do, however... I am going to actually shut that application down. Okay, now we're going to see the durability live in action. OK, so the app's actually running now so we can cancel it because it wasn't running before. So now we're actually going to cancel it. So that's what he's mimicking right now. So the reasons could be, you know, many, right? You know, the blade in a data center, you know, has a technical failure or some some digger, you know, wrecks the cable, all of them and you know, whatnot or simply a case is upgrading. You never know. But ultimately, things can happen and they will happen. Yeah, no, exactly. Failure is inevitable in so many of these distributed systems cases, right? Whether it is a Azure login scenario or there's a network glitch, whatever it might be, this could simulate your pod going down in Kubernetes or a rolling update, security patch by AKS. That's just a few that I can think of. Then if we restart it, so restart the application. Let's say we've repaired that cable. Yeah, repaired the cable. We've logged into Azure. So this is now going to start. And if I just switch back to the catalyst view, eventually we're going to see that workflow progressing through all the other remaining steps. And again, like from the kind of the background or the backbone perspective, like what's happened here is it made it to these subscription workflows, as well as, you know, if you click on one of those, Ochi and like the child workflows within them. And then each of those. Exactly. Each of those individual runs or activities was checkpointed into that state store for that resumption after the fact, ensuring that it can kind of run to completion. So, yeah. You probably saw the flick from running to completed, and if I switch back now, that we got that report, and that's the workflow done. Awesome. Do you want to show the email? Yes. Let's see if that ever comes. Ah, there you go. So for example, my email. So this is kind of a, obviously, sample email that the accounting people would get. But like I said, the CSV is kind of what we chose as a file format. So eventually, they would see this kind of a structure. And they could load it into their accounting systems any way they need. Yeah, awesome. So let's go back to the presentation. So yeah, I talked through a little bit of this already, but essentially like this kind of business logic is all wrapped within a set of workflow, durable execution wrapper. And specifically, you know, there's a number of other things you can do here. It's super modular in terms of supporting other patterns as well. So we kind of saw that fan in, fan out, meaning that you can do parallelization of a number of business processes, but then you can also do sequential workflows, child workflows, which we saw, which is again, spanning its kind of child process separately and then letting that run to completion. as well as things like waiting for external events, workflow versioning, a ton of those. This is also very interesting from an identity perspective. And this is something we're talking about a lot these days, especially with. agents and agentic workflows so being able to have an agent identity being able to lock that down from an access control list and being able to talk agent to agent in between these agentic workflows and last but not least on of retention and resiliency policies so right now this does not have a default resiliency policy built into it it just has that kind of basic checkpointing resumption but you can set resiliency and retry policies on an activity basis meaning that if you kind of Like, if that activity fails for whatever reason, you can have a specific behavior that you want to be executed, you know, whether it's retry, indefinitely retry once, never retry, whatever. And then this is just kind of a little diagram of how this is kind of working from the back end. I wanted to showcase this because I think it gives a really nice view into, you know, what we're actually doing within each of the Dapper sidecars and each of the Dapper processes. So like I said, it has a built-in Dapper... workflow engine into the sidecar and it's you taking advantage of a you know bring your own workflow state store and then as each activity is kind of progressing that is being written to that state store which is why oisin was able to kind of pick his application back up and then run that those workflows and child processes to completion and then it replays from the beginning. So every single time the execution state moves from the application to that Dapper engine, it's going to pick back up, run to the place in the workflow state store in which it's last checkpointed, and then continue on. Last but not least, I wanted to just touch a little bit in terms of, we've talked about two kind of stateful architecture patterns today, one of them specific to actors and one of them workflow. And it's an interesting, from an abstraction point of view, in terms of like workflow is just a higher level abstraction on top of the actor pattern. So if anyone has, you know, taken advantage of the actor pattern before, it's really powerful from a scalability perspective. So you can have, you know, these actors are sort of these individual unit. pieces of logic that have both state as well as you know they can communicate with each other and then an identity associated to them and you can run them at super high scale they play really nicely for a ton of scenarios you know we're using it here today to represent a cost center at unifar but you know you use it for many other applications as well. And then kind of workflows, what that does is it's another higher level abstraction, meaning it'll manage the state and the life cycle of those actors for you. And then even higher up, we have an agents framework as well for orchestrating sort of these multiple agent workflows and things like that. I guess just a little, one more slide in terms of lessons learned. We've worked really closely with Unifor on a lot of this stuff and wanted to kind of share some of the, you know, from the trenches. Absolutely, yeah. It's been actually a great experience. We did learn a lot. And like some of the stuff, for example, our very initial approach was just simply you write all your workflows and then you just, you know, start it, then let it run and just eventually comes back in. Then we realized, you know what, like as all these solutions kind of evolve, the number of resource groups, the number of subscriptions, the number of resources, all that's just going to grow. And so you don't really... you know you don't really have control in terms of you know part of the executions of all these kind of parent child worker hierarchy so we we adjusted a little bit so now for example we we run our subscriptions in parallel but then within the subscriptions we don't think sequentially that gives us that kind of a control like the the performance here isn't really in any shape or form a critical factor so that was for example one of the lessons we kind of when we were looking at all kind of frameworks even before kind of you know talking to to to to die grid we looked at kind of other frameworks and obviously you might recognize some code names like roslyn we eventually picked upper and for a few reasons we very much love the configurability so you know if you have certain components maybe your configuration localhost is different than the one in the cloud uh it's language agnostic which you know as much as we are kind of c-sharp you know house i mean you know certainly no harm and just the building blocks are there and it allows us to concentrate on our you know business logic without having to you know deal with these little nitty-gritty concerns and but we do actually extended certain things like one of our applications is is very kind of mission critical and so we wanted to have that kind of dapper deployment not just kind of within a cluster replication but also kind of regional so that we can run this across the globe but then certain contracts are still kept so we've done a little bit of an extension on that so that's been great experience and we've so far adopted dapper for a few of our other apps. So I'm just going to mention just a few of them as examples. We have this kind of thing we refer to as key rotation tool. Because not all kind of workloads can run under proper identity, legacy, or just completely different tech stack, or just deployment model that way. So they would use kind of key-based access to some resources. So we then have this tool where we can kind of on schedule rotate these keys or even on demand if we suspect some sort of security incident or maybe somebody shared the key inappropriately so we actually can do it kind of on demand and rotate. We have this other application that's very that that's the mission critical application I mentioned earlier we refer to it as front gate that's really where business people can configure some of these kind of legacy integrations generally file based you know SFTP some client upload stuff files to SFTP and then we can take them from there and move them to to somewhere else where SAP process can pick them up so That's the front gate. We're now starting looking into some of the agents. We have this idea about onboarding and offboarding resources, as in personnel, and licenses for them, et cetera. So we're very much looking into agents for that. And we, for example, found the workflows within the Microsoft agent framework to be somewhat alpha stage, and so we're very much looking into using dapper workflows for that. Awesome, and I think this is just kind of, again, an example of all the different applications you can use, common business processes modeled as workflows, which is, yeah, mostly up. So thank you. Thanks for bearing with us with the demo trouble. And this is all open source on GitHub. Yeah. Thank you, Oshin. So yeah, take a peek at the workflow, because it's all on there.

Don't miss out! Join us at our next KubeCon + CloudNativeCon events in Mumbai, India (18-19 June, 2026), Yokohama, Japan (29-30 July, 2026), and Shanghai, China (8-9 September, 2026). Connect with our current graduated, incubating, and sandbox projects as the community gathers to further the education and advancement of cloud native computing. Learn more at https://kubecon.io

Durable Execution in DevOps: How Uniphar Built Reliable Systems with Dapr - Alice Gibbons, Diagrid & Vaclav (Oisin) Haken, Uniphar

This session presents an end-user case study of how Uniphar, a global pharma and medtech manufacturing partner, uses Dapr to achieve durable execution across critical business systems. With Dapr’s Workflow and Actor APIs, Uniphar added a durability layer that ensures key processes complete reliably despite failures or restarts. Uniphar’s Mammon app uses Dapr Workflows to automate Azure cost allocation and stakeholder attribution, driving cost accountability in teams. Another app, KeyRotationTool, built with Dapr Actors, enhances security by automatically rotating keys through persistent actor reminders. Both apps are core pieces of Uniphar’s DevOps practices, with new resources being registered for rotation at deployment time, with their cost counted regularly in reports. Join Oisín and Alice to learn how Uniphar built fault-tolerant DevOps systems, the lessons-learned from production, and see a live demo of how the Dapr APIs implement durable execution in distributed systems.