Durable Terraform Applies with Temporal (Retries, State, Parallel Modules)

My name is Rob from RobOps and today we'll be looking at Temporal and how we can use it to make our Terraform executions more durable. Temporal is an open source platform that is capable of running long-lived reliable workflows in code. It does this by handling retries, capturing state, and supporting scheduling. In our use cases today, we'll be looking at how we can use these capabilities to help execute Terraform more reliably. We'll also see how we can use some of these features to improve how we operate our Terraform. As always, before we jump into the code, I think it's important to explain some important concepts about how Temporal works. Temporal runs workflows that we define. These workflows execute a series of activities. We define logic in our workflow about how Temporal should execute these activities. Workers will be responsible for running our workflows and managing their state. Temporal persists the full state and results of our workflows to enable retries, visibility, and external interactions over time. In our demos, we'll be showing how these features can benefit how we run and operate Terraform code. In our first use case, we'll show how Temporal can execute a parent workflow that will run child workflows containing Terraform modules. Here we can see how Terraform applies can be managed by a parent workflow and executed in parallel by child nodes. Before we jump into the code, I also want to mention that all of this code along with detailed readmes is available on my GitHub page that will be linked in the description. With that, let's hop into the code view. Now that we're in the code view, I want to take a minute to discuss how the code is organized. We have four working directories, one for activities, one for workflows, one for our terraform, and one for utilities. Now, if you open up the workflow directory, you'll notice we have a parent workflow, and a drift workflow. We also have Python files to start our respective workflows and we have our worker itself. Within the resources folder we have our child workflows and so these are the workflows that are going to be executed by our parent. In our Terraform directory we have a compute and VPC folder. These contain the Terraform that will bootstrap our networking infrastructure and our compute. And in our activities folder, we have our activities. In this case, they're Terraform activities that are going to run Terraform init, Terraform plan, and Terraform applies. Now, in our parent workflow, we'll quickly overview what the code itself is doing. Here, we define a decorator called workflow definition, which will define our parent workflow. Underneath it, we'll have a workflow run decorator that defines what will run when our parent is executed. Here we have the conditions of our parent workflow run. We have a maximum attempt of five and a maximum interval of every 10 seconds. These retry policies are highly customizable. Then we execute our child workflows. In the first execution we have our VPC result which is going to execute a child workflow called VPC workflow. We're going to pass it a payload, which in this case could be your TFVARs overrides or any variables that the workflow needs to run. We're going to pass it our retry policy, and we're going to also pass it some of the parameters that it would need to run, including a parent close policy. In a similar way, we will execute our compute workflow right after. As you can see, we can run as many child workflows as possible. In this case, we're going to be running them in series, but it's also possible to run them in parallel. Once the workflow is complete, it will return the results. In our worker file, we define how our worker should behave. In this case, we have to import the activities that we want our worker to be registered with. This means that we define what type of work our worker can pick up off of the temporal queue. And so in this case, we define our worker. And we say the worker should be attached to our client, which in this case is the temporal client we're running locally. It should only have access to the queue of my definition, which in this case is my task queue. And it can only execute the workflows that I provide it, as well as the activities. And so when we bring up this worker, it will be able to access our parent workflow along with the child workflows. And it will be able to access the activities that these workflows contain. So before we jump into the execution, I also want to highlight the temporal client that we have running locally. And so if I switch back over to my client, we can see that we have a temporal client running. And this is just a GUI that allows us to see all of the workflows that have been executed, their status, along with details about each of the executions. So if I click into the parent, I can see the last workflow that was run. So now we're going to see a workflow in action. So if I pull up my terminal, I'm going to start, first start a workflow. So to do that, I'll just run Python Workflow Start Workflow. I've started a workflow, but I haven't deployed any workers. And so this workflow is essentially going to sit idle until I deploy a worker. If I look back into my temporal UI and I click refresh, we're going to see that a Terraform parent workflow is running. but there's no workers registered currently. And so we'll essentially be in this waiting state. However, if I go back to my terminal and I switch my tabs and I deploy a worker, we're going to see the workflow instantly kick off and deploy our Terraform. So I'm going to click this command and hide my terminal. And we're going to see immediately that the VPC workflow comes online. If I exit out of the actual workflow detail, I'll notice that I've spawned an additional workflow of type VPC workflow. And if I click in here, I can see that Terraform has initialized, a Terraform plan has run, and it's currently attempting an apply. All of these workflows are managed by the parent. And so if I go back to the parent, we should see the VPC workflow complete and the compute workflow kick off. My VPC workflow is completed along with my compute workflow. In addition to this, I have my results. And if you minimize the results, you can see a very rich detail of everything that had happened during this execution. For my compute, we can see that I've applied Terraform to create one resource called an S3 bucket. If I look at my VPC activity, I can see I've applied nine resources. with the unique resources being route tables, internet gateways, and VPCs. I can also see the results of my init, which means Terraform initialization true for both my VPC workflow and my compute. If anything had happened along the execution path for this, we would have applied our retry policies. So if Terraform was trying to come up and it wasn't able to because something wasn't ready, it would have failed the apply. However, temporal would have continued to execute that Terraform and eventually become consistent with the defined state. Now that our infrastructure has been applied, we can quickly cross check this by checking our AWS. And I can see in my VPCs, I have a temporal VPC. And if I click into it, all of the resources that I created as part of this Terraform were applied successfully. Now, while that's all well and good, I want to also take it a step further. and see how can we use some of this information that temporal gives us to execute more complicated use cases and for that we're going to switch back to the slides and investigate something called drift detection so drift happens whenever we apply terraform whenever the state is different from what is actually applied in the infrastructure we'll get something called drift Drift is dangerous because when our infrastructure does not match what is defined in code, it could lead to confusion, overwriting, and potentially damaging our production infrastructure. And so in this next use case, I'm going to show how we can simulate drift detection using temporal workflows. And so for that, we're going to go back over to the code, and I'm going to describe a similar workflow called the drift workflow. And here we can see that it's a workflow run that's basically executing an activity called Terraform plan. However, instead of doing the full lifecycle, we're simply running this workflow over and over and over. And what this is going to do is it's going to constantly run a Terraform plan. And because we can capture the state of our runs, we're going to be able to see if and when drift has been detected. And more importantly, if drift is detected, we're able to use our own mechanisms to notify us about this drift and do something about it. In this case, we're simply going to log that drift was detected. But in your use case, you could use this to potentially reapply that infrastructure to realign it to the state. Or you could potentially send alerts to the relevant teams that care about this infrastructure. So with that said, I'm going to go back to my console view. and I'm going to pull up my terminal again and I'm going to go start drift. And so here, similarly, it's going to start my drift workflow. And so if I minimize my terminal and I refresh, I'm going to see that we have a drift detection workflow that started. Now I'm going to go back to my terminal and I'm going to switch tabs and I'm going to rerun my worker. Once again, we're going to see our Terraform plan activity come online. and succeed. And what I want to draw your attention to is back in the terminal. And so here we can see that the execution completed and we have the infolog no drift detected. Since we just deployed the infrastructure, it's exactly aligned to what we've defined. And as you can see in the background here, we're going to be waiting one minute before we run this plan activity again. Once the Terraform plan runs, we're going to verify that no drift was detected, and then we're going to manually inject drift directly into the console to show how our mechanism will detect it. So another plan kicked off, and if I go back to my terminal, as expected, no drift detected. And so what I'm going to do here is simply go look at the VPC and change it to VPC 22 and head back. Now that the plan is kicked off, we should see it succeed. And if I bring up my terminal, I can see instead of an info log, there's a warning log that drift has been detected against one of our resources. As we wrap up here, Temporal's ability to replace state unlocks the ability to manage how Terraform is executed. Our ability to implement retry policies allows us to execute Terraform more reliably. Executing Terraform in this way does make some things more difficult. such as managing outputs or passing variables between modules. However, drift detection is just one of many useful operations durable execution platforms enable. And I'd encourage you to check out the code base and propose your own unique use cases for how we can use Temporal to execute Terraform more reliably. And with that being said, thank you for watching. My name is Rob Ops, and if you enjoyed today's video or you enjoyed any of the contents on my channel, please, please consider liking and subscribing. If you're interested in learning more about the code itself, feel free to check out my GitHub page and start the repo. My name is Robert DiBolito. All of the code that we talked about in this video will be available on my GitHub page. With that being said, thanks for watching. Have a great week. Take care.

Terraform doesn’t have to be fragile.

In this video, we make Terraform applies durable using Temporal — an open-source workflow engine for long-running, reliable execution. We’ll use retries, persisted state, and scheduling to run Terraform more safely, recover from failures, and improve day-to-day operations.

✅ Code + detailed README:GitHub: https://github.com/robertdippolito/durable-terraform-workflow

What you’ll see:
* Temporal fundamentals: Workflows, Activities, Workers
* A parent workflow that triggers child workflows per Terraform module
* Running Terraform init / plan / apply with durable retries + visibility
* Parallel module execution to speed up applies and reduce operational pain

⏱️ Chapters:
00:00 Intro
00:51 Temporal Overview
01:53 Terraform Orchestration with Temporal
02:18 Code Overview
06:11 Orchestration Demo
09:07 Drift Detection Overview
10:43 Drift Detection Demo
12:36 Wrap up

💬 Question: What’s the worst terraform apply failure you’ve dealt with (and how did you recover)?

Inspiration: This video was inspired by a Medium article I read here: https://medium.com/@surajsub_68985/building-an-infrastructure-stack-using-temporal-terraform-on-aws-d2210b9f6582  — go check it out and drop the author a message.

#Terraform #Temporal #DevOps #PlatformEngineering #IaC