Actor & Workflow Improvements in Dapr 1 16

Well hello everyone, I'm Josh. I'm a maintainer of the Dapper project. I'm going to talk about some of the workflow and active improvements we made in the 1.16 release, particularly around performance and resource usage. So yeah, if you're not sure about workflows, workflows is an engine in Dapper which allows you to define kind of deterministic and durable. execution of business logic. It can take different paths based on inputs and outputs and ensures that that workflow durably executes until the end or errors or whatever else. But yeah, if your machine blows up from a meteor or something like this, then that will get, that workload will be scheduled on another node somewhere in your cluster and then continue executing. So yeah, that's what Firm Engine does. Again to go through very quickly, obviously we're in Dapper so you define your workflow in the programming language of your choice but of course now with multi-application workflows you can now do it in multiple languages and do it that way as well. We support things like fan out, fan in, external events, timers, child workflows, things like this. But going on to yeah so now I want to talk about an important note about how Workflow has implemented DAPA. So in DAPA we have the concept of actors which is a building block inside of DAPA and actors are a programming paradigm where you can define a single threaded execution of a particular process and you can have lots of them and they can talk to each other. Yeah you can kind of read more in the documentation about the actor model. But it's important to note that the workflow engine is built atop of the actor model and this is done for a number of reasons. So in the actor runtime we have the concept of this turn-based single execution of an actor. So a particular actor ID can only be executed once at a time in a particular place and that is very useful in the context of the workflow engine because when you have a activity that you want to run you want to ensure that that activity is only run on a particular host at a particular time and not multiple places. Of course, you're going to be using a workflow engine for critical systems such as payment processing and things like this. And you definitely don't want to be paid twice or send money twice, things like this. So the Actors model is useful for this. Also, it's useful for workload balancing. So the concept in DAPA of actor placement. So we use the placement service to distribute actor IDs over a natural distribution across all of the replicas and so with using the workflow engine on top of actors we get a natural distribution of the workflow execution across all the replicas for a particular workflow and then finally identity based isolation so workflows have identity they have a name associated with them as well as an instance id and an execution id and each of the activities are named and have these ids and these all get fed into the actor system where we have an actor ID that represents all of these instance IDs and so on. And it's important to kind of bring this up as a lot of the performance improvements that we made to the workflow engine and also kind of tie into the actor subsystem as well. Great, so we made workflows stable and that was great, but in this release we focused a lot on the performance improvements. So I'm going a little bit into the kind of nitty gritty details here, which might be interesting for you. And obviously, DAPA the the runtime is written in go so these are kind of a lot of a lot of these are kind of um go specific things but kind of generally uh kind of software engineering kind of stuff as well um so there's a lot of things we made changes to was reducing memory allocation so reusing data structures across and reusing those data structures across different actors. In Go we have this kind of nice utility called sync.pool which allows you to once you're finished with a data structure that you don't need to use anymore you can give it to this kind of pool of data structures or same or different types and that you can pull from that pool at a later time without having to go back to the operating system and say give me more memory. We can kind of use the reuse that memory. So we reduce allocations. We re-architected a lot of the actual runtime to reduce those allocations generally. So moving a lot of the shared memory up into the system, which reduced things a lot. We reduced contentions. So specifically, when I talk about contentions, that is where it goes back to the kind of term-based actor kind of behavior I was talking about before, whereby... instead of getting to a particular stop in the code and then having to wait to get to your term we've re-architected things so it reduces all of those contentions so removing things like new text in favor of just kind of continuing on and then also having a concept of this kind of optimistic getting of memory so instead of having new text around maps and things like this we use called a synch.map which optimistically basically tries to write or read from that map and then if it didn't win then backing up as it were. So kind of reduces the amount of time that the program will be kind of locked and waiting which is useful to kind of keep things going and also reducing context switching and things like this which can you know lead to performance degradation. We reduced a lot of the Go routines are used as well throughout the runtime in favor of using callbacks and this is helpful as well because this means that the although Go has a very great system with with go routines and the um uh the kind of orchestrator for the go routine runtime inside of go very good once you get up to something like a million go routines at the same time the performance degradation kind of isn't worth the benefit right like it doesn't really make sense to have to have any go routines and we say we've removed a lot of those as well and we've also uh made some changes to the way that the dapper talks to the scheduler uh scheduler service we have all streams open now which helps with bursting and general through things like this so a lot of kind of smaller kind of nitty-gritty details but once you kind of start adding them all together you see these kind of a much better performance numbers generally but also a better performance profile coming on to now so for example this is running these two slides are running for on an EKS cluster. So we're running a three node EKS cluster with T2 medium sized nodes, I believe. And they will be running three replicas and they're executing a thousand workflows with 15 activities that are kind of doing kind of operations. The main thing is, you know, running workflows with those activities and we're scheduling more at the same time and then waiting for them to finish. The top one is 115 and the bottom one is 116. And you can see here with the memory difference, you can see that the 116 has a, by definition, uses less memory. You can see it peaks at the top here at 130 across the three replicas. And at the bottom one, they all peak at 90. What's also important to note is that they peak at the same place, which means that we have a much better memory distribution across our replicas. So we're using... close to the amount of same memory which is a good performance profile to have. This helps with with with replicate bucketing. Also they are a lot more stable and this is much better for designing how much memory you're going to need so capacity planning and it's much better when you have a kind of stable profile like this so we have a much more memory profile stable system and it is a lot more consistent across the replicas so we're doing much better at utilising all of the kind of resources across the replicas in the same amount of way. And we go on to CPU, and it's much the same story here. We're using less CPU by default. You can see here it kind of peaks out at 0.48, 0.25, so we've been a lot better here. And again, it's consistent in the much closer together, which is much better for kind of bucketing and capacity planning scenario. And then finally, there's a big graph here showing different scenarios. Less is better, and you can see glue is across the board lower. and as you can imagine as we're going to get further up in scale with a number of workflows running at the same time and or replicas you can see the bigger difference in the performance improvements we've made and we can see here that some of the orange bars aren't even present on this graph because the the system just conked out and can handle it and crashed out so now we're doing a lot better now with stability with these kind of larger kind of workflow throughput scenarios which is great Of course, the main takeaway is we've got less resource consumption across the board. That resource consumption is much more stable, particularly at scale. And we've reduced the workflow time to completion. So specifically, going back to this graph, this is time to completion of the workflows. And I've got some links there as well. You can look at the release notes that we had for 1.16, which goes into this with more detail. And yeah, the workflow for OOP.com. I've added there as well to give a give yourself a bit of a read on on workflows generally. With that, I will hand it back. Thanks a lot, Josh. Great to see all these detailed diagrams. Cool.

Dapr maintainer Josh van Leeuwen presents the details behind the actor and workflow enhancements introduced in the Dapr v1.16 release. Josh shares runtime-level optimizations that significantly improve performance, stability, and scalability of workflow execution.

Josh covers:
 • Durable, language-agnostic workflows (Go, C#, Java, Python, JavaScript)
 • Actor-based execution model ensuring deterministic orchestration and workload distribution
 • Memory management improvements through sync.Pool reuse
 • Reduced contention by restructuring shared memory and adopting optimistic sync.Map operations
 • Lower overhead through reduced goroutine usage and callback-based execution
 • Increased throughput by expanding scheduler streams

Performance highlights from benchmarking:
 • Up to 30% reduction in memory consumption with more even distribution across replicas
 • Lower and more stable CPU utilization, simplifying capacity planning
 • Faster workflow completion times under load
 • Improved stability at high concurrency, where previous versions would fail

These improvements make workflows in Dapr 1.16 more efficient, predictable, and reliable at production scale.

Further resources:
 • Dapr v1.16 Release Notes → https://blog.dapr.io/posts/2025/09/16/dapr-v1.16-is-now-available/
 • Workflow Overview → https://docs.dapr.io/developing-applications/building-blocks/workflow/workflow-overview/