Simplifying distributed systems development with Temporal's durable execution

As Ben mentioned, my name is Tom Wheeler. I'm a principal developer advocate at Temporal. I've been with the company for about three and a half years, although I've spent about half my career as a software engineer, and I've dealt with many of the problems that Temporal solves. And that's really what I want to begin my talk today, explaining to you what that is. So if you've never heard of Temporal before, you're not sure exactly what it is, it's a natural question. What is Temporal? I want to actually start from a different direction. I want to explain why Temporal exists. There's a few trends that have happened over the last 10, 15, 20 years. If you think back a generation ago, every corporation had its own data center. All the computers lived there. Applications didn't really scale. like they do today things would run on a single computer uh maybe a small set of computers and those computers are right next to each other it's easy for them to talk we did our logic through making function calls but as we needed to scale we need to spread things across multiple computers and function calls became rpcs we were invoking operations on other machines and our infrastructure moved to the cloud developers started adopting microservices because you have these big monolithic applications that you deployed as a single unit. And there's a lot of risk with that. And anytime there's risk, the velocity slows down because you could have a change in one part of the application that breaks something in an entirely different part of the application. And splitting things into microservices... allowed us to allocate resources to the parts that need it. You could give this part more memory. You could give more CPUs to this thing over here. You could have more computers running a particular service. It also gave us flexibility as developers because we could implement each thing in the language of our choice and use whatever database we wanted on the back end. So there's a lot of benefits of moving to microservices, but these two trends have together made pretty much every modern application a distributed system. There's a lot more potential for scalability, but there's also a lot more opportunities for failure because instead of invoking logic within a single computer, we're invoking logic on other computers, and we're doing that over a network. The other computer could be down. Your network connection could be down. The network connection somewhere along that route could be down, and those are the source of a lot of problems that we deal with. Effectively, we're all distributed systems. developers now it used to be a generation ago that was more of a niche area of computers but everybody solves the sort of problems that i've described now so i'm going to give you an example of something that you know an application that you may never have thought much about the implementation for an expense report at some point you've probably submitted an expense report you create the report you put in your receipts you put in your expenses you keep doing that some point you click submit your manager gets a notification they review it they approve it or reject it maybe it gets sent back to you to make some changes if it's rejected otherwise it goes to accounting they reimburse it you get a notification for approval so this involves a lot of different systems that have to integrate together and they all have to be running for this thing to take place successfully If you were implementing this, you'd probably have an application database to store the information. You might have message queues to have different parts of the system communicate with one another. You might have a cron or scheduler that is sending notifications. So you have a lot of pieces of infrastructure. And all of those have to be up and running and functioning the way that you expect. Also, this involves multiple people. It's a mix of automated and human-in-the-loop sort of things, manual processes. And it also, the entire process could take multiple days, even multiple weeks from end to end. And during that time, things could crash, things could fail. And so you've got to make sure that your applications are reliable. So let's drill down into just one part of that, the reimbursement. A reimbursement might seem like a single operation, but it's actually two operations. You've got to take the money out of the company account and put it into the employee's account. Both those things have to happen, and they both have to happen exactly once. So this is a distributed system. Here I have a function to do that reimbursement, and it creates a client for one bank's API, the employer, another client for the employee's bank, and then it does a withdrawal, it does a deposit, and then it generates a confirmation message. This seems so easy, right? Six lines of code. How hard could it be? Well, this is the joy of being in development where you don't have to solve real world problems. But let's take a descent into the madness. So the problem that you're going to encounter first probably is that you can't make a connection to one of those banks because either the service down or the network is down. So you say, I've got it. I'll implement support for retries. And I'll just keep making requests until one of them finally goes through. So then your code goes from six lines. It's now quadrupled in size, but now you can survive an outage of the network or the service. The next thing you might encounter is that the service returns an error. Maybe you were able to withdraw the money and you go to deposit the money and it gets rejected or something occurs, you can't deposit the money. So now the account balances are off. You've got to compensate by putting the money back where it came from, effectively rolling back the transaction. When you add support for that using the saga pattern, which is a common way of undoing things in a distributed system, now your code gets even bigger and more complex. Another problem you might encounter is that you successfully make a request, but you never get a response back. Well, you can't wait around forever. You've got to, at some point, fail that request and try again or perhaps call a different system or something or alert a human that they need to take action. the next thing you do is, okay, I'll add support for timeouts and then fail the request and don't get a timely response. So you do that and now your code looks like this. So maybe you can recognize this. Maybe this has happened to you. Nobody sets out to write code that looks like this. Everybody sets out to write code that looks like this. But as you get into production and you encounter problems and find solutions for them, your code becomes more complex. So let's just compare the before and after. we started off simple as we got into production we encountered problems we tried to mitigate those problems and now our code is really complex and you think okay we've solved all the problems everything's cool this application runs reliably but does it no there's still a big problem here so uh i'm just showing kind of an excerpt of that larger code sample but what if you make the you do the withdrawal but before you make the deposit the application crashes. And it could crash due to a defect in your code, a bug in a library dependency, a kernel panic in the operating system, a power outage, hardware failure. There's a lot of things that could cause this code to crash in the middle of processing. And it might seem like a one in a million thing, but I bet all of those things have happened to you at one time or another. And when you're operating at a certain scale, these things happen all the time. Everything fails all the time. So... If the code crashes between these steps, you've got a problem, and it's a hard problem to solve. So I want to introduce a term, durable execution. Has anybody ever heard of that term? A few of you? Okay. So I'm going to do a demo here, and what I'm going to show you, I think the best way to understand the concept of durable execution is to compare normal execution to durable execution. And I'm going to do that by way of a very simple program that I wrote in Java. It's not terribly involved code. It's probably... One of the simplest Java programs you could write. I just got a class that's got a main method that calls this method called count2. I pass in a number. And so in the implementation of count2, I just got a for loop. It's going to count up to that. And in each iteration, it's going to print out the current number. So we've got some application state there. That current count is the state of our application. And then it's going to sleep for one second. And in Java, you know the api is that you specify the duration in milliseconds and i just want to mention what an unusual thing that is or what a notable thing that is for decades i've been doing java for a very long time and for decades the only way that you could specify time was in milliseconds like maybe 10 or 15 years ago they added support for nanoseconds so you could get even shorter amounts of time but thread.sleep was never meant to sleep for like a day or a month You know, that would be crazy. You can't expect the program you start now is going to be running a month from now. So I just want to point out, you know, like having long running programs like this that pause for long durations is not something that the language designers intended. And the same is true for a lot of other languages. So I'm going to run this now. And it's going to print out one, two, and so on. It gets to the end. Okay. So now I'm going to do it again. And I'm going to kill it. It printed out two. So when I rerun it, what number is it going to display next? One. Right. You know that because you are developers. You understand how computers work. But if you ask somebody that doesn't know anything about computers and you said, what's the next number that would come out? They might say three. That's what you would expect to happen if you don't know anything about computers. They don't know about the loss of application state that takes place when a program terminates unexpectedly. But I'm here to tell you that they can work that way. and so let me show you that that is durable execution so compile and run it prints out one two i kill it start it up again takes a moment to recover the state three four five and it prints it out and so that very simple idea the fact that you can kill a program and then have it resume from where it left off rather than start over from the beginning changes your world as a developer and so let me talk more about that Durable execution is crash-proof execution. It's made possible by an abstraction that's provided by a durable execution platform. And Temporal is an example of a durable execution platform. In Temporal, that abstraction is called a workflow. And I want you to set aside any baggage you have about the term workflow. You might be thinking about drawing little pictures of, you know, what your code should do. In Temporal, you define workflows through code. We support Java, Go, PHP, Python, Ruby, C Sharp. I may be leaving out a language. The community has created an SDK for Swift. There's one for Clojure. So there's a lot of different languages you can write this in. And you can actually mix and match those languages together in a single application. So you can have Java that calls into Ruby, for example, or vice versa. So a workflow is nothing more than a function or a method, depending on the language, that has durable execution. Once it begins execution, it's guaranteed to continue running. even if it's killed it's able to sort of reconstruct the state and then resume the execution and continue on so let's look at what's happening behind the scenes in that early example this is the non-durable execution the normal execution the computer starts up we print out one we print out two bang i killed it that's where it ends the story ends right here the process terminates execution came to an end That's because normal execution is bound to a single process that's running on a single machine. And when that process exits, we lose the application state. We lose the program counters. We no longer know which things have already run nor what's left to run. With durable execution, it essentially virtualizes this so that execution can span a series of processes, each of which could potentially run on a different machine. And that means you can withstand hardware failure as well. So here's what it looks like with durable execution. We start up. One, two, three, program dies just like it did before, but that's not the end of the story with durable execution. Process number one terminated before the third step completed, but it resumes in a new process, reconstructs the state, and then continues on from there. So it prints out three. Let's say it dies again. And this time, let's say it dies due to a hardware failure. So what happens now? Reconstruct the state on a different machine in a new process. Then it continues, prints out four, prints out five, and at this point execution is complete. So the summary is we, in this example, we began in process one, we made a little bit of progress. In process two, we made some more progress, and then we finally completed in process number three. So we have an execution that spans multiple processes over time. The physical representation of what took place here is below the line. Sure, it ran in three different processes, but from your perspective as a developer, it doesn't matter. And the reason it doesn't matter is because this is what the developer experience is. Everything ran. You don't even see or know or worry about the failures. I mean, you can find them if you want, but they don't affect you. And because they don't affect you, you don't have to write a bunch of code to handle all this. You don't have to write code to manage application state. You don't have to write code to manage... uh what happens in in case of a crash it all happens automatically and transparently to you as a developer so durable execution i think is a revolutionary idea it's a really simple concept that your program can survive these catastrophes but the impact is profound because it makes your application more reliable that's obvious but what's less obvious is that it allows you to focus on the goal You focus on the happy path. You code for success, what you want your program to do, instead of spending all your time anticipating and trying to mitigate all the problems that could happen along the way. As a result, you have applications that are faster to develop and easier to maintain because your business logic is not surrounded by this complexity of error handling and state management and things like that. So what is Temporal? It's an open source, durable execution platform. It automatically maintains the state of your application. It enables your applications to recover from crashes. as if they never even happen. So you don't have to worry about them as a developer in the same way that you don't have to worry about, you know, what order do packets arrive in on the network? You know, the operating system takes care of that, abstracts all that away from you. And the same way that Temporal abstracts away crashes and things like that. It also offers support for retries and timeouts. So that complexity of the code that I showed before, when a service goes down or the network goes down, you have those benefits automatically. so you can survive failures and outages of services that you're calling into ultimately it improves developer productivity by making your applications easier to develop scale and support so now i want to show you a demo another demo this time it'll be with strike oh by the way let me introduce the temporal web ui the temporal web ui allows you to see every execution that takes place you can see the ones that are running right now you can see ones that have run in the past and you can know exactly what happened at runtime so i'll just show you the me counting to five so you can see right there it's in the timeline view print one and sleep print two and sleep then there's this delay because i killed it right after two it reconstructed the state and now print three and sleep print four and sleep print five and sleep and down in this table we can see everything that took place Information in this table will tell us, for example, like what machine this ran on, exactly what time it started, what time it finished. We have all the information we need if we need to debug something. So that's very helpful. Okay, so this is on Temporal's community GitHub. There's a scooter demo there. And I'm going to run that. This is an application that uses Temporal. So it's written in TypeScript. And there's three parts of it. There's a back end, which is the temporal code, an API, and then a front end. And the front end knows nothing about temporal. The front end calls to this API, and the API calls temporal. So all the code for the temporal stuff, let's just see here, there's only a few hundred lines of code, a lot of which don't even have anything to do with temporal. So if we're just to look at the workflow itself, it's 150 lines of code. in the workflow itself so this the workflow is doing all the orchestration for what i'm going to show you and uh think about like a lime scooter or bird scooter or something like that so i'm going to log in and unlock the scooter and it's going to start the ride and so now i'm riding through town and it's racking up if you look in the lower left you can see it's using these tokens this is using stripes usage-based billing so in my temporal code I'm making calls to Stripe's APIs and I'm doing billing and I'm charging based on how far someone's riding and how long they're riding and things like that. And so let's just say we're going to end the ride now. And there we go. And if we look at the workflow in the web UI, we can see everything that took place. We found the Stripe customer ID. We begin the ride. We add some distance. Charge for that we post the distance charge to stripe same thing for we're adding time charges as well And so we can see everything that took place all along the way. Let me show you another example here and We'll dismiss that and I'm going to choose scooter one two two three four Maria example calm and so now We're trying to start the ride, and it says, oh, ride initialization is taking a while. Please wait a moment. And if we go into the web UI, and look what's happening, we see this red line here, and this red line will tell us exactly what's the deal. We can see network error while attempting to contact Stripe. Now, Stripe's not really down. Stripe is a very reliable service. This is something I did in a demo to simulate a network error. But if you're calling some other service that's not as reliable as Stripe, if you're trying to query a database and that database is offline, if you're trying to write to a file, the file system's full, you may encounter some other problem. Well, in this case, it's just retrying. It's automatically retrying that call. And we can see from that number right there, four, it tried four times. And on the fourth time, it was successful. And so now if we switch back, then we can ride. So we have this network outage. and it didn't wreck the program there was a delay but now we're able to ride show you another example here real quick and let's say we're going to put in a test at example.com which is not there's no account associated with this so we try it and uh in this case the workflow that manages the ride fails we wanted it to fail because this is an invalid writer so we have control we can make things uh fail when we want to but when that happens we can see right exactly what happened that the uh tells us exactly what line of code happened you know what took place and they have everything we need uh one last example and i'm going to use one two three nine put in a valid email address Okay, so now if I go back to the web UI, there's another one running. And this is failing. Let's see what's going on. So if I just click this, it'll tell us exactly, oh, invalid scooter ID, 1239. Wonder why that's invalid. Well, let's go look at activities.ts line 56.3. I'm going to kill the back end for a moment in this very terminal. I'll go look at activities. And right here, I can see there's a bug. It's got a regular expression that's trying to check the scooter ID. And instead of doing 0 through 9, it does 0 through 8. So it's a typo. So I can comment that line out. And I'll put in that one, which has the correct regular expression. So I fixed a bug in my code. This has nothing to do with temporal, but I fixed a bug in the logic of my application. And now I'll just start that back end again. So I killed the program, to be clear. And now I can ride. So if we look back at the history, scroll up, close this there, we can see that sort of red and green line, those red lines are telling us that we had a problem at that moment, but it got resolved and now everything is happening after that. So I'm able, what I just demonstrated is I fixed a bug in live running code. And you can do that because of this durable execution. The program was running, but it had a problem. And when I fixed that problem, it retried and the problem, you know, it skipped over that because the problem's not there anymore and it's able to continue successfully. So that's Temporal at a glance. Again, it's a durable execution platform. It's open source. It has support for all the common languages that people typically use. And it has this web UI that gives you observability into what your application is doing at all times. And finally, I didn't show here in Stripe, there's the usage-based billing. So these are the tokens that I've consumed. It's all showing up right there in the Stripe dashboard. So you can connect to Stripe and any of the services that they provide APIs through or any other service you want to use. That's all stuff you can do from a Tempor workflow. You just write the code in whatever language you want. and you get the reliability and the developer productivity that I described.

Tom Wheeler, Principal Developer Advocate at Temporal, explains how modern distributed systems and microservices have made every developer a distributed systems developer - and why that's a problem. With a live real-time demo using Temporal and Stripe's usage-based billing API, he shows how Temporal's durable execution platform solves the reliability challenges of distributed applications, including network timeouts, retries, and state management.

*Links*
learn.temporal.com
temporal.com

*Markers*
00:00 - Introduction and why Temporal exists
01:06 - The evolution to distributed systems and microservices  
01:52 - Problems with distributed systems: the expense report example
04:04 - The reimbursement problem: from 6 lines to complex code
07:06 - What happens when your application crashes mid-process
07:19 - Introduction to durable execution concept
08:49 - Live demo: normal execution vs durable execution
10:03 - How durable execution works behind the scenes
13:19 - The developer benefits of durable execution
14:01 - What is Temporal: open source durable execution platform
14:59 - Temporal web UI walkthrough
15:54 - Scooter demo: real-world application example
18:11 - Handling network failures and automatic retries
19:49 - Debugging failed workflows with detailed error information
20:59 - Live bug fixing in running code
22:24 - Conclusion