What is Temporal? Durable Execution Explained

Hey y'all, DataGuy here and today I have a video for you where I wanted to cover one of the hotter new tools in the workflow orchestration space and that is Temporal. Temporal has come out to kind of solve the problem of hey you know if you're building a system, something like an order fulfillment system you know and a user places an order and then code needs to charge a credit card, reserve inventory, notify a warehouse, send a confirmation email and update analytics, there's five potential failure points and so what temporal does is says hey we are going to actually make sure all of those are going to complete perfectly and you have that guarantee that everything is going to complete exactly how your code was designed to run through a term called an open source durable execution platform so the execution platform part starts with you know hey Temporal is not just a library or a framework you add to your existing code. It's a runtime that manages how your code runs and what your code, not what your code does. And it takes responsibility for persisting your execution state. And durable means your code's progress survives failure. So there's no try to recover it. this will be recovered no matter what. If a workflow starts, its state is recorded to persistent storage at every step. So that if the server running your code crashes, Temporal can just replay the record history and your code resumes from the exact point it left off. So that's what we're going to talk about today, go into how Temporal works, why it's kind of exploded in popularity and now reached a $5 billion valuation. And then I'll show you how you can get started using Temporal on your local machine. So without further ado, let's get into it. So let's talk about why Temporal has gotten so popular. And the major reason is it kind of represents a paradigm shift of, you know, instead of you needing to design your application code to be responsible for tolerating failures, the execution environment itself is. So you don't need to worry about building all of that into your code. Temporal handles that recovery. And it also means that your code keeps running, even if there's a server crash, there's a network partition and timeout, there's a deployment restart, there's a cloud provider outage, there's a failure that spans days or weeks. We've all seen with all the outages recently, things like these are really important. so critical systems don't become interrupted or lose their perfect state and redo work. And Temporal actually started inside Uber as an internal system called Cadence because it had to handle hundreds of critical business processes across Uber's microservices architecture and then went open source with Temporal and now it's been in continuous production for almost nine years. And the key facts here are, hey, it can run in pretty much any language that you need. you think about using except for Rust, Python, TypeScript, Joe, Java, Net, PHP, Ruby. It's deployment model is typically, you know, you, hey, you can deploy the temporal server, which is a sub-process your workers connect to, and you own your code and wherever it's going to run, and temporal just manages the execution state. And that gives you that core guarantee that, hey, you know, a workflow that starts will eventually complete. It's never going to silently disappear in a queue. It's never going to get stuck in an unknown state. And it also, kind of inverts the responsibility for reliability because in the traditional model you had to design application code that was responsible for retrying failed operations and tracking progress through multi-steps and you know handling partial failures but now you don't have to worry about doing any of that your applications code describes what needs to happen and temporal takes the charge of ensuring it why you know it actually happens and so you can think of it as a runtime that makes distributed code as reliable as single process code the same way you know a database might make distributed data as reliable as data on a single disk and so the core primitives that you can see here in this diagram are are such you have first the workflow which is your business logic and this is just a regular function in the language of your choice and temporal will record every step that workflow takes into a event history. And so if the server dies mid-execution, the workflow replays from the event log and will pick up exactly where it left off. So your code will look like normal sequential code and then under the hood, every time an execute activity call is checkpointed, there's going to be, you know, the log of that. And so a crash between any two steps means that Temporal can replay that history. Then within a workflow, you have, you know, your activities. And so activities are where your side effects happen, you know, an API call, a database, right? A file IO, sending an email and activities are expected to happen, right? That's the point. And then temporal gives you the implicit capability that they'll be retried with automatically with configurable back off if they fail. And you can also configure things like retry behavior per activity. And then over here we have the workers and the workers are the process that you pull run that pulls temporal task queues and then executes your workflow and activity code. So you own this process. It's your application code that's just connected to temporal. And this is basically just saying, hey, I'm looking at your task queues, looking at what's happening and understanding, you know, making those checkpoints, relaying it back to this application. And so. If your temporal server never loses your code, runs your code, it's only managing your state. Your workers are actually running your code. And the separation is why this system is resilient, because the workers can be scaled and restarted and replaced, but the workflow state is tracked in a separate location. And then within this, you also have things like signals, which are sending data from the running workflow from the outside, like a user canceled the order. You have queries, which read the current state of a running workflow without interrupting it. And timers have workflows that run across. work across restarts because they'll just restart that workflow. And so where Temporal is going to be best used for, right, is in scenarios that are really long running processes with multiple steps and really real world failure services. So some examples that are, you know, the one I already used, which is order fulfillment, you know, multi-step competing transactions, payment processing, where you need exactly one semantic so people don't get double charged. customer onboarding, maybe if you have multiple flows, spending email verification provisioning, subscription billing, where you can have your month long sleep timers and renewal logic and AI agent orchestration as well. There's been a lot of people that are using AI agent orchestration for really, really long running. LLM loops because you know that allows you to pick up from where it left off so if you're running a days-long process and it fails in the middle you don't have to run the entire day back again and it's not the best fit though when you need sub millisecond latency because temporal does that overhead per step also if your logic is stateless and simple you know a single API call doesn't need temporal and there is also a bit of a learning curve so if you're not ready for the mental model shift you'll might not be the best fit or right now And then finally, kind of just, you know, What are the advantages and disadvantages of terminal here are, you know, number one, you're eliminating any kind of dead letter queue. You are, you know, there's no pulling cron jobs. There's a ton of infrastructure that you can remove because you don't, you know, it's just completely replaced by temporal. You also get visibility out of the box. You know, the temporal web UI gives you a real time view. It will show you if every running workflow, its current step and its history. So you don't need to add additional observability tooling to your application code. Also has really good correctness. So exactly once activity execution, things like that mean that you can get distributed systems correctness without having to build distributed systems yourself, which as everyone knows is very difficult. Also language native code, there's no external orchestration language or structure you need to use and it's open source as well. Disadvantages though is that workflow code needs to be deterministic. Same inputs must always produce the same execution path and replay. So no random, no direct date time now, no direct network calls from workflows, which is the kind of mental model shift that might trip up newer users. It also is, if you're running it self-hosted, you need to run your own temporal cluster, which requires Mandarin Cassandra or Postgres and a temporal server and front end and matching services and production infrastructure. cloud is what they're kind of pushing you to, which eliminates a lot of that complexity, but adds cost. You also have event history size limitations. Every workflow execution has max history size of 50,000 events, which obviously can be a lot for a lot of workflows, but could also be very little for some workflows. And very long running workflows with thousands of iterations might need to reset that history periodically, which kind of loses some of the benefits. Also, cold start latency, right, because every activity execution involves a round trip to that temporal server. If you have workflows that have hundreds of steps, this is going to add up. And so temporal is more designed for correctness of the tradeoff, you know, against raw throughput. And also the replay based execution model might be unfamiliar to developers. And, you know, there's kind of bugs that can be caused by non deterministic code. So just, you know, we'll take some kind of massaging of existing code bases to actually have it work right with temporal. So now how do you get started using temporal locally first thing just brew install temporal I've already done it so you know just gonna see that and then you just want to run you know temporal dash dash version and so for old version make sure it's actually installed properly and Boom, we should all be good and it's very fine temporal real quick. So just give it a second. Awesome. We're chillin And then we just run temporal start dev and this will start our local temporal server, which if we want to go to the UI, we just click here, follow this link. And now we have our temporal web UI here. And so this is where we can see all of our different workflows and you can see they actually have different samples. So if I want to go, you know, check out some go code from temporal IO, I can go download it here and start running my own workflow like that. So now that we've got a temporal server running, I want to show you how you can just set up your own workflow with temporal. So first what we'll do is just create a kind of little demo environment in this folder. So make directory temp demo CD in there. And then we are going to do a quick virtual environment. And then we are going to activate our good old virtual environment. And then we are going to pip install just in portal.io and HTTPX. And then we are going to create a... activities.py in here we're going to define our temporal activities so like I said you know when you're defining workflow you're just defining hey what are the activities we want to actually run so we're reporting async io temporal io as activity and then we're just defining these activities using the temporal framework of hey simulate a slow external call so just creating an account or you know calling an email service here right and these are just dummy activities so I can simulate this and create a workflow out of them So then after we've got our activities defined, we'll create workflows.py. And then what we can do here is import a lot of the same policy. So temporal IO workflow, time, delta, date, time, and retry policy. And so here we're importing our activities. And also make sure these are all in the right folder. So let me just do that real quick. So here we're importing activities and we got imports passed through. So greet user, send welcome email, and you got to use a special workflow safe import. And then we have our user onboarding workflow here. So we're just running our retry policy. So just notice, hey, this is the retry policy we're setting for this workflow. And then we're generating our greeting and just executing these two different activities in a workflow. So We've got our workflow now. And then last step we're going to create here, or not the last step, second last step is create our worker.py. And so this is what's going to actually run our worker and connect to our application and monitor activities. So here we are importing temporal client, worker, our workflows and our activities, connecting to that local host. And then we're connecting to this worker to that temporal client. So, hey, we want you to monitor this user onboarding workflow for the activities greet user and send welcome email. And then we're just, you know, kind of looping application here. And then what we'll do is run this worker in a new terminal. So here, if I go and create a new terminal window here, and then run into CD temp demo, and then I want to activate. Let's run this worker here. And one second, make sure this is all installed. So just add install.temporal.io in my default environment. So now I got my worker running here for my temporal workloads. And then last step is starting a workflow execution. And so here, what we're going to do is create one last Python file. So here we're going to call this run.py. And we're going to import async IO, temporal IO client, the user onboarding workflow, then connecting to that local host, executing this workflow. And now what this workflow is going to do when we run it is just run this workflow and we'll monitor it within temporal and make sure that it's actually succeeded. So go to our terminal down here and just so you guys can see it here, clear this and then Python run dot PI. And now if we. see this workflow completed hey boom our workflow is our account has been created so if we go back and open up our local host let's see everything you can see now we have our workflow the event history so we have greeting user send welcome email and the entire event log for that event for that whole workflow And you can see here the relationships there's none the worker word actually executed on call stack pending activities And then also you can see any scheduled workflows batch operations different namespaces But that is for another time. I just want to show you how you can get started with the basics of temporal I O In 150 minutes, so I hope you enjoyed this video. I hope you found it helpful Hope you have a great rest of your day day guy out

In this 15-minute crash course, we dive deep into Temporal.io and how it eliminates the "distributed systems tax." Learn how to replace messy cron jobs, dead letter queues, and custom retry logic with durable execution.

If you’re building backend services, data pipelines, or orchestrating multi-step API calls, handling failures is usually the hardest part. Today, we break down exactly what Temporal is, how its architecture works under the hood, and walk through a complete local setup using Python.

🚀 Check Out My Long-form Data/AI Courses: https://whop.com/the-data-guy-llc/

Use Code dataguysub for 25% off!

🚀 Get Source Code and Bonus Content: https://patreon.com/TheDataGuy?utm_medium=unknown&utm_source=join_link&utm_campaign=creatorshare_creator&utm_content=copyLink

⚡ Follow my Substack: https://substack.com/@thedataguygeorge

🎬 Watch My Daily Data & AI Shorts: https://www.youtube.com/@DataandAIGuyShortForm