From Legacy to Durable: Migrating Enterprise Workflows to Temporal

All right, so quick intro about me. I'm Ian. You can find me on our Temporal community Slack as just Ian Douglas. And I'm iandouglas736 on most other social channels if you want to connect and ask questions. I'm part of our developer advocacy team here at Temporal, and you can find me at events and doing webinars and things like this. So for today's agenda, here we go. We're going to look at how we use Temporal for enterprise needs with your architecture in mind and what considerations you're going to need to plan for ahead of time. Next, when you see that Temporal is the right tool, how do you move the systems that you already have running in production onto Temporal without breaking something that works today? And then we'll end the session with some Q&A. If you're not already familiar with Temporal, I want to cover some quick high-level overview of what we offer as a technology. Temporal lets you write applications where functions run across crashes, deploys, restarts, and network outages without losing state. Workers replay every event from history to rebuild that state, and that's why event history is going to matter so much later on as we talk through this. We have a self-hosted option, which you can operate in your cluster, you can manage your own database, the visibility store, your auth layer, and so on. We also have Temporal Cloud, where Temporal operates all of those things for you. Either way, you run workers and clients within your own infrastructure and execute your own code, and we help manage the state and retries and so on for you. So quick note before we get into it, because this will affect how you should read older advice that you may find. If you see other guides on how to move to temporal, check the date of when it was written. We've added a lot of new functionality in the past year and can make some things much easier and even faster to move to temporal. So if you've read an older guide and came away thinking like, oh, moving to temporal means building a lot of things, some of those things are built into temporal now. So let's jump in. We're going to talk about reliability, auditing, handling things at scale from an architecture point of view, and how you can start to shift these over to temporal. When it comes to reliability, your workflows need to survive crashing at any point, period. You might have written a lot of code yourself to keep things running, like what if that API goes down? Do we fail over to another API? Do we need a circuit breaker pattern to handle downstream errors from happening? Or do we sit and hope a service comes back online before everyone starts getting notifications? Temporal brings all of this on its own, whether the problem is infrastructure going down like an availability zone or region, or if a process crashes unexpectedly because some user input failed to get validated properly. You're not writing retry code anymore and a lot less recovery code. Auditing is a big one. If you caught my office hours last month, I talked about how to use temporal and regulated environments where audit trails are a really big concern. With temporal, every step is recorded as an event, and you can use search attributes to write queries to scan through everything, which can matter a lot if you're doing any sort of incident response. The key here is that whether you're building for regulated environments or not, like SOC 2 compliance, HIPAA, GDPR, You should always be planning an immutable log so that you know what happened, when, maybe by who, and in a way that you can't tamper with. For scale, it impacts a lot of our businesses at an enterprise level. As workers scale out, you can separate your workloads across multiple task queues. And if you have a multi-tenant system, you can also use priority and fairness, which will ensure that all tasks get an opportunity to execute. With all three of these things, reliability, auditing, and scale, if you think about everything you have in place, just to keep some fragile process up and running whether it's a dead letter queue some kind of recovery or reconciliation service or pulling loops for stuck workloads that's all stuff that you can start to retire from your code base with temporal and fewer moving parts means fewer things to monitor and hopefully means fewer notifications if something goes wrong there are two different pieces in the architecture the control plane is what we call the temporal cluster It holds your event history, your task queues, and the visibility storage. You can use that as temporal cloud, which is managed for you, or you can run the open source version yourself in production. And you can certainly use the open source version locally for development. The other piece that I call the compute plane is your workers. These run on your own infrastructure, your virtual private cloud, or VPC. The workers pull work from the control plane on those task queues and run your workflow and activity code on your infrastructure. we don't see or execute any of your code your business logic and your data stay within your own environment with temporal you're handing off the hard distributed systems part keeping the state durable scheduling retries visibility but you're not handing over your code or your customer data durable execution is most useful for work that runs for a long time at temporal that genuinely means that you can run things for days months or even years Some examples could be things like a customer onboarding that's waiting for them to upload a document, or a claim that's paused for someone to review with something we call human in the loop. Or maybe you're running like infrastructure as a code to start and stop new systems that get created, scaled, used for a while, and then torn back down. The pattern to remember here is what we call entity workflow. You have one long-running workflow per real thing. so one workflow per customer per claim per service instance that you're launching it sits in a loop it waits for work it handles one thing at a time and runs a health check now and then we have a blog post about this where we tell people to design your systems for the long haul and i'll describe this a little bit more what does it mean to design a system for the long haul well in practice it comes down to three habits you want to keep your state from growing forever You want to use continue as new to reset your history at sensible checkpoints. And you want to keep your activities item potent. If you get those three things working correctly right up front, one of these workflows could run for years if that makes sense for your business. One thing to note is that Temporal is an at least once system. We help you recover from things, but it's still up to your code to ensure that something can only happen once, even if a worker retries a piece of that workload. We'll make sure it always happens, but if a retry could duplicate the outcome, like a new charge on a credit card or a new row in a database, that's still on you. Item potency is a big thing to keep in mind when building good architecture, and you're going to hear me say that quite a lot. Earlier, I mentioned something called human in the loop, so I want to give you a quick note on that. Almost every business process has a person involved somewhere, an approval, a manual fix, maybe an exception that needs intervention before continuing. So we have three ways that you can talk to a workflow that's already running. First, we have signals, and these push data in, but it doesn't wait for a response. It's like fire and forget. A query reads the current state, but it doesn't actually change anything. And then we have updates. This is a newer one, and a lot of older material you might find about migrating to temporal may not cover this yet. Updates check the request. changes the state, and hands back a result all in one call. Most simply put, a query is like a peak, a signal is a push, and an update is a pop. So one rule to be careful about here is that validators for queries and updates only read what's going on with the state of things. Any code that you write for a query or update should not block or wait on an activity or you're going to end up in a deadlock. So here's an example. If you have a workflow running, and it hits something that a retry can't fix, like bad user input, instead of failing, it can pause, mark itself as being blocked, so someone on your team can find that with a query, and then you could have a system set up that sends in a correction using a signal with a corrected input maybe, and then the activity can pick up where it left off instead of crashing and having to start all over again. I also mentioned earlier that activities run at least once by default. and temporal retries them with exponential back off so the right move is to let temporal handle the temporary failures instead of giving up on the first problem the catch with at least once and it's the number one mistake that we see is that if your activity isn't idempotent that retry can charge a card twice send a second email or create a duplicate record in the database so any activity that has a side effect needs an item potency key tied to something stable like an order ID, an invoice number, a unique customer ID, that sort of thing. And we recommend using UUID values. Two more things. We want to be deliberate about what kinds of errors you can retry. If it's a timeout or a rate limit, absolutely retry those. But if it's something like bad input from your user or an auth failure or a business rule violation, you can't really retry those. So you're not retrying something that's never going to work. You should also assume that any activity code could run again after a worker crashes, even if that worker actually finished its work, but maybe crashes before it got to a report back. So idempotency is really important. And this is exactly the thing that bites people once they're running old code and new code side by side. So you need to plan for this as you get started. Here's an example of some code that you might have in the system right now. we see a retry loop around a payment api that sometimes fails you're doing the retry you're managing the back off by hand you're doubling that wait time you're capping it you're hoping you don't double charge the credit card because item potency is all on you and if this process does need to give up we have to push something into what we call a dead letter queue which means you've also built some kind of alerting system some kind of reconciler service to drain that queue maybe check if you need to refund somebody and that you didn't reserve too many resources or some other cleanup process. It can be a lot to manage. Plus, if that process crashes in the middle for any reason, you lose all the progress unless you're also tracking the state of things in your own cache or database along the way. With Temporal, that becomes much simpler, and we change our charge method here to just calling the API and handling the result. The difference is that all retry behavior moved out of the code that charges the card. And we move it into a policy now. We do the first interval, the back off, the maximum attempts, which errors don't retry. And that whole thing is durable. So if it crashes, it picks back up where it left off. And now the retry, the back off, the cap, giving up, those are now configuration options instead of actual lines of code. And now you get to clean up all that other code, the dead letter queue, any reconciliation or cleanup you might have needed before. And this is what a temporal workflow looks like. The bit of code at the bottom here would run as a workflow that maybe does multiple activities as part of an entire workflow. Now, a common pattern that we see from people that start on temporal is that they wrap every single activity into its own workflow, and then they have a ton of workflows going on, and you don't have to. Typically, a workflow would cover many activity steps, like a user registers, and they subscribe to something, and you handle the billing, and so on. And that would all be one workflow. But if you have an activity which is just one durable step, like sending an email, handling a webhook, syncing a database record, charging a credit card, you can build that as a single activity and have a workflow then that orchestrates many of those activities. But here's something that makes getting started with Temporal even easier. You don't need to rearrange a bunch of code to be a whole workflow definition with all of the activities set up. Here on the screen, we're setting up something called a standalone activity. It's the same activity code from the last slide. The only difference here is that you're starting this directly from your app code in the same place instead of setting up that workflow. So you still get durability, retries, timeouts, all the visibility. You just skip kind of that workflow overhead, which if you're using temporal cloud also means fewer billable actions, but you also get lower latency. You'll notice on the second line here where I'm setting up the activity ID. I'm tying this to an order ID so the call is addressable and it gets deduplicated for free. Now here's the best part that we really want to stress when you're migrating and why this is the best way to get started. This is about the smallest first step that you can take. And when that one step eventually grows into a multi-step process, the same activity just becomes a step inside a workflow. There's no huge rewrite. One caveat today is that standalone activities are in public preview and pre-release for most of our language SDKs. They're not generally available yet, but they will be in the near future. Okay, let's talk about determinism. It's kind of a big word around temporal, so I want to talk about that for a moment too because it can trip people up. Temporal recovers a workflow by replaying its code against the recorded history of events in the exact same order. If you deploy workflow code where you add or rearrange activities in that workflow, Temporal can't recover, and it's going to throw something we call a non-determinism error, and the workflow is going to stop. So the rule here is kind of strict, but it's simple. Your workflow code has to be deterministic. Remember that your workflow code and your activity code are separate things. Your workflow has many activities. Workflows will record the values from their activities and store that as the workflow state and reuse those values if a replay happens during recovery. Your activities, which should be anything with a side effect like we talked about, other than or any other kind of like random occurrence like network and database, input, output, file access, things that runs in threads, those are allowed to be non-deterministic because the result of that activity gets recorded back at the workflow level. And that gets replayed from history. So a good rule here is to have your workflow code contain things like branch logic for which activities to run. and try to move as much of that operational code into activities. If you keep that boundary clean, then determinism errors generally don't happen. I mentioned versioning a minute ago, and that's generally something we're familiar with in software development. One thing that worries people most about running long-lived workflows on Temporal is how do I ship new code if I have workflows that started weeks ago and they're still running? And the answer is worker versioning. You can run more than one version of your worker. which are all watching on the same task queue, and each worker is tagged with a build ID. And the server is going to send each task to the right version of the worker. So workflows that start on version 1 stay on version 1 workers, and any retries or replays happen there, any new workflows start on version 2. Nothing replays across those versions. We're never going to replay a version 1 workflow on a version 2 worker. But you do need to keep your version one workers running until your version one workflows are finished. Now, if you're on Kubernetes, the worker controller handles the whole lifecycle through a custom resource definition or CRD to bring up the new version, drain the old one as its workflows finish, and then handling the cleanup. We have a blog post that we'll add in the notes about safe deployments on Kubernetes if you want more ideas and help with that. Two more tools for the really long-lived cases. First, when you actually need to change the logic inside a running workflow, not just deploy a new worker, but you have to introduce a new step or change the order of the steps. Temporal has a patching API call, and it's called GetVersion. This is going to let the old and new paths coexist until the old workflows finish. And then for a full rewrite, you just make the new workflow type, call it version 2, and you run them alongside each other. But the downside of patching is that on a workflow that lives for a really long time, those patches kind of pile up and the code starts to get hard to follow. We have a command to send workflows, or sorry, we have a command to send two workflows. If you want to continue an old workflow, but start with a fresh history, we call it continue as new. To handle this whole patched code piling up thing, we introduce something called upgrade on continue as new. So when you call the continue as new, you can also tell it to upgrade everything. So now each individual run finishes on a single version, while the workflow as a whole moves across versions over time at the continue as new sort of boundary. It's a good fit for entity workflows, long running agents, and things like batch jobs where your business needs have to change a bit over time. And this is in public preview right now. Now the safety net that's kind of under all of this is that you should add replay testing in your continuous integration pipeline. Basically, if you replay real saved histories against your new code before you deploy, you can actually prove that a change isn't going to break anything that's already running. All right, so that's a lot to throw at you. So I want to give you a little recap here before we go into some Q&A. How do we even get started with all of this? The good news is that the starting point is smaller than you think. You start a local dev server within a couple of minutes. You pick one workload that already kind of hurts, that flaky retry loop, the queue consumer that's constantly sending notifications to somebody, and you wrap up just that bit of code as an activity. Same code you already have, but now it's durable, it's retried, observable, and it's behind one worker on one task queue. You run that worker in your own environment in a separate terminal to prove that it's working, and then you're done. It's going to surprise you how quickly you can get started with this. Once it's ready for bigger testing, you can deploy Temporal on your hosted environment or VPC. You can then deploy your code to point it at the new control plane, start your worker, and that's it. Now you have a real production workload. Now, if you don't want to set up Temporal on your own infrastructure, you can get a Temporal Cloud account, and we manage all that state information and queues and so on for you. And to make it easier, you can install the Temporal Developer skill. It gets your coding agent using determinism, item potency, versioning correctly right from the first piece of code. And it's always going to use our latest code examples and documentation, so you're always up to date. And here's the honest follow-up. What do you actually have to change in your architecture to keep this migration going? Well, it feels like a long list. There's seven things here that you need to track. First and foremost, you're splitting your code into logical workflows of activities. You need to have deterministic workflow code that orchestrates and activities that do all the work that has the side effects. If you get that part right, most of the rest of this list is going to follow automatically. But second, item potency is not optional. Every side effect from your activity code has to be keyed to a business ID because retries and replays cannot change the outcome. Third, big state tracking that you might have, it needs to go away. pass references to object storage instead of passing large blobs of content fourth you've got a new thing to deploy you have a set of workers and those need to run in your vpc or you can use serverless if you use aws lambda that that talk to the control plane which is temporal and then fifth your deploy process should change worker versioning and replay testing should happen in your ci pipeline in order to block deploys once that's working your deploy process doesn't have to stop killing other processes or whatever in order to ship new code. Sixth, you can map out your task queues to separate workloads and tenants and with priority and fairness where that matters. And last, number seven, a fair amount of logic that you run today like cleanup processes for dead letter queues, cron jobs, hand rolled state machines, they can all go away in favor of the built-in pieces that Temporal helps you manage. It's a bit of a mind shift that changes how you build and how you think about your architecture. All right, we're almost done, but here's your migration playbook. How do we grow from there? We've started with that one activity, but where do we go from there? So you have to think about what needs to migrate first. You want to find the kinds of processes that are holding state, running for a long time, and are kind of a pain to restart and recover where you have to fix something by hand. This will pay off the fastest. If you already have a good fire and forget system like a broadcast on Kafka or something, leave that alone for now. If it's boring and reliable already, don't migrate that yet. Secondly, you want to wrap things incrementally. Start with standalone activities if you're comfortable with preview mode code, or start with a small workflow that only has one or two activities. Find the smallest pieces with the least amount of rewrite. Third, plan how your systems are going to coexist. Every risk in running new and old code together has a specific temporal feature. We're going to look at that next. For example, you could look at Nexus if you have multiple workflows on different business teams that don't need awareness of how they each work. They just need to work. Versioning can help with deploys and so on. You're not building the plumbing anymore. We're going to cover this more on the next slide. And number four needs a moment to explain. When we talk about pain, times fit. This is a way of coming up with like a scoring system for what to migrate. Something that's really high paying that fits well into temporal is a great candidate to get started. So find the thing that annoys you most or some kind of cleanup process that you hate to run or how high the compliance exposure is if it breaks. The fit is how well suited it is for durable execution. Does it hold state? Does it run a long time? Does it have multiple steps? Does it fail in a way that you can recover? Those are high fit. So we want to look for a high score on high pain as well as high fit. That's why we kind of multiply these together to come up with a score. Lots of pain, but a low fit, like a broadcast path, it's not really a good use case for temporal. You could migrate it, but you won't see a strong benefit. So find something that's a good match to temporal and then monitor how well that handles things and use that to find the next thing that you want to migrate. Just basically start with the one thing that's going to be obvious that it worked. So here are things broken down a little clearer. These are pain points or risks you might look at today and what to use at Temporal to solve that case. You can go ahead and grab a screenshot of this if you want to go look that stuff up now. We're going to share the slides and some FAQs and other documentation with the recording in a couple of days. So we don't really cover Nexus too much in this session. We're going to have an office hours about it in the future. You can also check our YouTube channel for some talks that we did at our replay conference last month. Basically, in a nutshell, Nexus gives you the same durability but across teams and namespace boundaries so your old systems and your new temporal services can call each other without sharing a code base or a deployment. This can matter a lot with migrations because typically different teams move at different speeds and you can't make everything kind of cut over all at one time. All right, last slide. And I'm going to leave this one up for a moment as a recap of the recap. You want to find a small piece, migrate it, watch it work, migrate from there. You want to measure the changes, like how much less you get notified of something breaking or how much time you're not spending maintaining code or processes anymore, and use that to plan the next thing to migrate. And go install the temporal developer skill for your teams so they get the latest and greatest help right where they do their development. All right, we're going to pause here for some Q&A. We did have one question come in. Can workers be serverless? The workers can be serverless. and those serverless workers can cover both standalone activities as well as workflow activities we've got some good talks from replay conference that you can find on our youtube channel and you can also go look at our documentation for that and we'll share those links in the faq document that we're going to send out with the slides but yeah you can go check out the temporal documentation on serverless and it'll tell you how to get both of those set up Someone else asked a question with queue based systems today like Kafka, SQS and RabbitMQ. Do we throw that out? You don't have to necessarily throw them out, but you can reevaluate what those workers are actually doing. So as they're draining the queue, like what are they doing? Are they doing activity code? Are you is your like what's pushing into the queue and what's draining the queue? That could be a good case for, hey, what we're putting into a queue. That can be our workflow where we have multiple steps. and then what's draining the queue those are your activities and now you can migrate all of those over to temporal and temporal is going to manage that queue for you at scale on temporal cloud if you host your own you'll probably still need to manage that to some degree on your own but with temporal cloud we'll manage the size of those queues and we'll scale those queues as much as you need someone else asked a question a process may hit a permanent failure what happens to partial progress Again, if you have a workflow kind of defined where a workflow has multiple activities, the workflow is basically storing the response from each of those activities. And if an activity fails, the workflow is basically going to go through and replay, and it's going to replay all of those values. And then whichever activity did not run, it'll kind of resume at that point. And so it kind of replays the history until it gets to the activity that didn't work. And then it'll retry that activity until it gets a response and then it'll move on to the next activity and so on. And then it looks like last question we had, how do we avoid double processing during a dual run? That's a great question. And it really comes down to item potency. And that's really, it's a little bit, you can think of it like a little bit like a race condition where you've got maybe multiple processes that are trying to charge a credit card and which one wins. If you have an item potent uh item potency key then your system needs to be built in a way where you're saying okay this item potency key is about to charge the credit card and if this other one is coming in and it's waiting it might get blocked on that item potency key until the first one gets a response and if that response doesn't come back then you can have some kind of timeout that says okay that first process didn't finish go ahead and let that second one through again that would be up to your system Typically, you would want to try to avoid a double run of that same system, though, because like we described earlier, temporal is run at least once, not exactly once, at least once. So we're going to make sure that that thing is going to try to happen. But you really only need one worker handling that thing one at a time. And then once it gets a response back, it'll finish from there. And again, if it doesn't get a response back, that workflow kind of halts at that point and you can it'll. basically retry based on the policy that you set up. By default, we're going to do exponential back off, but you can adjust that for however you like. You can set a cap on how many retries. You can set a cap on the total duration to wait. So you can say back off every two seconds, try a maximum of 100 times or, you know, 10 minutes, whichever comes first, and it'll basically stop at that point. And then again, you can look at why did that fail? You can maybe send a signal in if you need to adjust something and have it retry after that. All right. I think that was all the questions that we had come in. So thank you to Hannah for helping moderate that. And thanks for hanging out today. Yeah, if you have other questions, you can always reach out to our Slack community. We'll get the recording as well as a whole FAQ guide for you on how to do these migrations. It'll cover a lot of notes. It's going to give you lots of links to documentation. um we'll send out a link to our slack community if you have additional questions you can always reach out to us there we love helping people over there and we'll catch everybody in next month's office hours thanks for coming

Getting started with Temporal is the easy part. The hard part is what comes next: your organization has years (or decades) of business-critical logic living in brittle, undocumented, deeply entangled, and potentially unsupported systems. Cron jobs that nobody dares touch. Home-grown state machines held together with hope and tribal knowledge. Automation pipelines built on tools that were already outdated when your senior engineers started.

This session is for teams that have working knowledge of Temporal, and are now facing the real migration challenge. We cover practical strategies for incrementally wrapping legacy systems with Temporal workflows without requiring a full rewrite, how to identify which processes are highest-value migration targets, and how to manage the coexistence period when old and new systems have to run side by side. 

Whether your legacy is a message queue, a customer orchestrator, or something that would make a conference room gasp, this session has something for you.

Key takeaways
- Learn how to identify which legacy processes are the highest-value migration targets (and which ones to leave alone for now)
- Understand strategies for incrementally wrapping legacy systems with Temporal workflows without requiring a full rewrite or big-bang cutover
- Know how to manage the coexistence period when old and new systems have to run in parallel without introducing new failure modes
- Leave with a practical framework for building internal alignment around migration priorities and sequencing

---

Temporal is the simple, scalable, open source way to write and run reliable cloud applications. 

Learn more
Blog: https://temporal.io/blog
How Temporal Works: https://temporal.io/how-temporal-works
Community Slack: https://temporal.io/slack

Developer resources
Docs: https://docs.temporal.io
Courses: https://learn.temporal.io/courses
Support forum: https://community.temporal.io