AWS Lambda Durable Functions

Hello everyone, today we'll be covering AWS Lambda Durable Functions. This is a new feature from AWS that lets you build long-running workflows on Lambda with automatic checkpoints, free waiting and automatic retries when something fails. In this session we will cover four main topics. What serverless is, the problems with standard Lambda, and how this new feature solves those issues. We are going to run a real demo, so let's start it all over. Let's start with serverless. With traditional servers, you provision them, patch them, and scale them manually. But with serverless, you upload the code and AWS handles everything else. The pricing model is simple. You pay by the millisecond of execution, no traffic, no cost. It also scales automatically from zero to millions of requests without any configuration. That's why it's called serverless, not because there are no servers, but because you never have to worry about them. And now how we got here. Before Lambda, developers had to do everything by hand. It started with virtual machines. You pay for the server, you run it, and you fix it. Everything was your job. Then came containers, less work, but you still managed the infrastructure. Then came Lambda, no servers, no setup. You write the code and AWS handles the rest. And now let's introduce our new feature, Durable Functions. It is still Lambda, but without any time limits and no lost progress. Every single step makes the developer's job much easier. Now let's see how the Lambda system actually functions in practice. Lambda is like a worker who sleeps until something happens, like a user opening your app, a file being uploaded, or a message coming in. So Lambda wakes up, does the work, and goes back to sleep. While it's sleeping, you pay nothing, of course. You only pay when it's actually running. Okay, now the first time Lambda wakes up, it needs a moment to get ready. We call that a cold start, but from the second time on, it wakes up fast. However, this workout follows a specific rule. It will only work for exactly 15 minutes. Once the time passes, it returns to sleep, even if the work is not yet finished. Now, let's look at the issues with standard Lambda. Lambda is great for quick tasks, but when you're building long-running workflows, four limitations emerge. The first problem is the 15 minute timeout. Suppose your workflow takes 30 minutes to complete. Lambda just cuts it off right at 15. The whole job stops completely and all of our progress is just gone. Second problem, Lambda has no memory. Every execution starts fresh. If step three fails, Lambda forgets step one and step two. You end up writing code to manually say progress after every single step. And the third problem is that you pay while you wait. Your workflow pauses for external approval, which might take two hours. Lambda bills you for every second of that wait, even though nothing is actually running. And the final problem is that orchestration becomes complex. You end up with multiple Lambda functions, database queues, and event routes all connected together. A simple workflow turns into a distributed system that's hard to debug. These are not just simple edge cases. This is what happens when you scale Lambda in a production environment. Durable functions solve all four of these problems one by one. The 15 minute limit is no longer an issue. Each step still runs inside a standard Lambda, but the entire workflow can run for up to a year. memory issues fixed progress is saved automatically after every step if something fails the workflow resumes from the last save point rather than starting over paying while waiting fixed that too the workflow can pause for hours or even days and while it's waiting you don't pay a thing Zero. Complex code fixed as well. You write everything in one function, like in Python, JavaScript or Java. No extra services, no complicated setup. Every single problem that we just talked about, they are all solved now. And now let me show you how this works in a real-world scenario. A five-step food delivery order. Step one, check if the restaurant is open. Step two, charge the customer. Step three, send a confirmation message. and then the workflow pauses to wait for the kitchen. This wait doesn't cost anything either. Step four, assign a driver. Step five, confirm delivery. Now imagine that step four fails. The driver system is completely down. With standard Lambda, everything restarts. You charge the customer again. You send the message again. It's a big problem. With the new Durable Functions feature, steps 1, 2 and 3 are already saved. Only step 4 retries. Junior says once the driver system is back online, the workflow automatically resumes from the exact point where it previously left off. Consequently, no data is lost and there are no redundant or repeated functions. So this example clearly illustrates why so many teams are switching to Durable Functions. Now, Allow me to briefly summarize the key differences for you. Let's compare them side by side. With standard Lambda, you have a 15 minute limit. With durable functions, workflows can run for up to a year. Standard Lambda loses everything upon failure, while durable functions automatically save your progress and resume from where they left off. Standard Lambda charges you while it's waiting. Durable functions cost nothing when idle. Standard Lambda functions require multiple services to be orchestrated together. Durable functions, on the other hand, let you write everything in a single function. This completely solves the four problems we just covered. Let's jump into the demo. Let me show you what we're building before we jump into the code. An order comes in and it hits the orchestrator. This is our main function that controls the entire workflow. The orchestrator runs five steps in sequence. Step one, verify the restaurant is open. Step two, charge the customer. And step three, set a confirmation message. Now for the interesting part, free wait. The workflow pauses for 30 minutes. During this time, Lambda is completely shut down. Billing stops. You pay nothing. After the wait, the workflow resumes. Step 4, assign a driver. And step 5, confirm the delivery is complete. So if step 4 fails, for example, let's say the driver's system is down. We don't lose anything. Steps 1. two and three are already checkpointed only step four retires so that's the architecture now let's take a look at the code all right here's the code one file with everything in one place so let me walk you through it we start with the imports the decorator from the aws stk durable execution and durable step Here is our first step, Verify restaurant. Notice the decorator. It's marked as a durable step. This acts as a checkpoint. Once finished, AWS automatically saves the result. And the same pattern applies to the other steps, like charge customer and send confirmation. Each one is a durable step. Each one is a checkpoint. And this URL serves as the courier API endpoint for our fourth step. For this demonstration, it has been configured to fail intermittently by returning a 503 error. The step is designed to automatically retry the operation. And now, assign driver. This one calls the courier API, and if it fails, it retries automatically. Now here is the orchestrator. the decorator, I mean durable execution, marks this as the main function, the one that controls the entire workflow. Here, context.step, this is how we call the checkpoint function. Please take note of the name parameter, that is exactly how we identify each individual checkpoint, and also this specific context, please wait for 30 seconds. It's free while waiting as shown in the diagram. So lambda shuts down here and the billing stops. I set it to 30 seconds for the demo. In production, this could be hours or maybe days. When the time limit is up, the workflow will automatically resume its process. Once the wait is over, we resume the assigned driver process and confirm the delivery. And finally, we return the complete order status, which includes all of the necessary transaction details. Alright, now that we have finished reviewing the code and the entire workflow, we can finally proceed to deploy our function to the AWS Cloud Platform. AWS provides us with three ways to create our function, either from scratch, meaning you do everything from A to Z, which is what we did in the code we saw earlier, or we can use a blueprint, which consists of common AWS use cases that are pre-built templates that we can review and then enhance or customize, but we don't require this at the moment, or we could choose a container image. If a function requires specific libraries or packages to be present in the instance before the function starts, then instead of just uploading the source code, we can upload the entire container image containing the libraries, packages, and the source code. But we are going to go with building it from scratch, which is our preferred approach. We start by defining the function name and then we define the runtime environment. We'll just go with Python. Moving to custom settings, we'll walk through each one. First, let's look at the EC2 capacity provider configuration. Lambda is serverless, meaning we don't manage the underlying EC2 instances ourselves. But AWS just released a new feature, the EC2 capacity provider. If a function requires specific hardware, you can create that EC2 instance. When you create the function, you specify which EC2 instance it should run on. This is actually the opposite of the serverless concept, but AWS has made this feature available now. We are currently using a custom execution role for this setup. When a function first runs, it only has basic permissions to write logs to CloudWatch. If we need to connect to a database or require specific access to DynamoDB, we can create and assign custom IAM roles. However, we do not require that at this moment. Now we also have the VPC where I want this function to run inside my private VPC. However, I don't really need that feature right now. When the function is initially created, we can have AWS automatically assign it a unique URL. For instance, if I happen to have a webhook, I can easily access it through that specific URL. Now we're going to move on to the core feature, which is durable execution. We have two configurations and these are very important to us, especially since we were talking about durable execution. Now the execution timeout is how long the workflow can keep running. We mentioned that it can stay active for up to a whole year, so we can set it to 360 days here. The retention period is... how long AWS will keep the logs in the console. It's set to 14 days here, so after 14 days it will delete the logs from the console. Now I'll revert to the default setting, I'll finish it later. Hey, 30 minutes and the retention period is 14 days. And what we'll do... Okay, after you create the function, we notice in the watch that AWS has already created a file which is just a hello. Well, you see, its handler configuration is also currently pointing to this file. So we just need to modify the handler to point to the main file name, which is actually defined as app.lambda handler. We are currently deleting the old file and we perform an update. Now, by way of this update, or perhaps we could say we do, Uploading our files can be done via a zip archive or we can host our source code on S3. And for this demonstration I will proceed with the first method which is the zip file approach. And then you just click save. This is the very same piece of code that we were looking at earlier today. We are finally ready to put it to the test at long last. And here we should have the happy scenario because our backend is currently returning 200. So we notice here in the catching preparer, it will succeed now, but it will wait for 30 seconds. And these 30 seconds, as we mentioned before, we won't pay anything for them at all because Lambda durable execution functions charge based on the actual execution time. Exactly. This is the happy scenario because our backend is currently returning 200. We can see the output. Exactly. It returned just as we expected. Perfect. All right. Perfect. We have tested the happy scenario and everything went well and worked as expected. Now we'll simulate assigning a driver. It appears that the backend infrastructure responsible for the driver assignment service is currently throwing a 503 error code, which clearly indicates that the server is temporarily unavailable to handle our incoming requests. Alright, let's take a deep breath, focus our full attention, and dive right into the complex process of troubleshooting and fixing this specific issue. We will also notice that the kitchen, prepare kitchen, will remain started and wait for 30 seconds. As we mentioned before, we won't be charged for these 30 seconds at all. Here, as we noticed, the sign driver has failed. We do not want to run a simulation, we just need it to return the data from its backend. So we wait and we'll notice that only the sign driver will be repeated and the rest of the steps will remain as they are. Then it performs retrace exactly as we observed. It is also entirely possible for us to go ahead and check the system logs. Right now, we are actively routing all of our function logs over to CloudWatch. As you will notice, when the operation failed, it logged an error. To us, even as we're collecting the info, as you can see, we can perform aggregation at the logs level and identify the root cause from it. All right, after seeing this path, when the sign driver backend returned 503, what could happen? We could check the monitorings. We've enabled CloudWatch metrics so we can see the invocations, duration and error counts here. But there are some very important metrics we need to look at together, which is the most expensive invocation in gigabyte seconds. As we can see, the 30 seconds it takes to prepare the dish are not... we aren't being charged for that at all and the maximum duration we were charged for is about five seconds. This confirms that the new feature from AWS Durable Functions only charges us for the actual execution time.

Welcome to Cloud Softway's deep dive into AWS Lambda Durable Functions! In this session, we explore how to build long-running, stateful, and fault-tolerant serverless workflows without stitching together a maze of Lambdas, DynamoDB tables, and EventBridge rules just to keep a single process alive.

When you build workflows on standard Lambda, you hit a wall fast. Functions are killed at the 15-minute mark, state vanishes after every invocation, you pay for every second spent waiting on an API or a human approval, and orchestrating anything non-trivial means writing fragile glue code across multiple services. We discuss the real cost of these limitations, workflows split unnaturally across functions, state manually persisted in DynamoDB, compute billed during idle waits, and brittle multi-service wiring that's hard to debug and maintain. To solve this, we demonstrate how AWS Lambda Durable Functions turn a single code-first function into a checkpointed, resumable, long-running workflow engine.

In this video, we cover:

* The Serverless Journey: How we got from VMs, to containers, to Lambda, and now to Durable Functions, and why "no server management, pay per execution, auto scaling" still left a gap for stateful, long-running work.
* The Problem with Standard Lambda: The 15-minute hard limit, no state between calls, paying while idle, and the complex glue code needed to orchestrate real workflows.
* What Durable Functions Bring: Workflows that run up to a year, automatic checkpoints that resume on failure, free waits with no compute charges while paused, automatic retries of only the failed step, built-in idempotency, and a code-first model in JS/TS, Python, and Java, with no state machine DSL to learn.
* Standard Lambda vs Durable Functions: A side-by-side on max execution, failure behavior, billing during waits, state management, and code complexity.

Live Demo: See Durable Functions in action, a long-running workflow that checkpoints at each step, suspends for free while waiting, recovers cleanly from a mid-execution failure by resuming from the last checkpoint, and runs the whole thing as a single, readable function.

Learn how Durable Functions cut the cost of idle wait time to zero, eliminate the fragile glue code and manual state handling that slow teams down, and make complex, mission-critical workflows dramatically simpler to build, operate, and audit all on serverless.

Learn More: Built for businesses that outgrow their infrastructure. Discover more about how Cloud Softway designs and operates scalable, cost-optimized serverless architectures at: https://www.cloudsoftway.com

#AWSLambda #DurableFunctions #Serverless #AWS #CloudComputing #LambdaWorkflows #EventDriven #StepFunctions #CloudArchitecture #DevOps #CloudNative #Idempotency #FaultTolerance #CloudEngineering #CloudSoftway