PAL 102: Orchestration & Observation – Understand Workflow State and Guard Against Failure

Hey again everyone, my name is Bianca and I am a customer success engineer here at Prefect. Welcome back to PAL. This is module 102. If you haven't watched the video for module 101, I would recommend doing that first before watching this video all the way through because a lot of the material that's covered in the first video will prepare you for what we're going to discuss in this session. If you want to follow along with the GitHub examples at home to complete the lab, there will be a link to the GitHub repo down below, so we'll be sure to update that for you as the course evolves. If there is a Prefect Engineer that is guiding you through the PAL program, then they will be sure that you get the updated GitHub repository so that you can run all of the examples in your own environment. And with that... Let's go ahead and get started. We're going to kick things off here by covering the agenda for this module. We're going to start by looking at tasks, another prefect concept in more detail. Then we're going to segue into logging for observation, runtime context, states, retries, variables, blocks. prefect integration libraries and a bunch more helpful resources for you to leverage while you are exploring prefect so it's going to be jam-packed with good stuff let's get to it let's kick things off with tasks how do we go about writing a prefect task well All we need to do is add a task decorator to a Python function to enable a host of orchestration capabilities such as retries, caching, and async convenience. We're going to cover an example over the next bit of slides where we're going to see Prefect Tasks in action. So we're going to have a three-part pipeline. It's going to have a function that fetches weather data and returns it. it's going to save the data to a csv and return a success message and then last but not least we're going to have a assembly function that calls the first and second function in order here is our fetch data function this may look familiar if you already looked at module 101 this is our fetch weather flow that is retrieving weather data from the open medio api We're going to expand our script by adding a save data function. What this is going to do is it's going to take the information that we got from our fetch weather function and store that weather information into a CSV and then print out a successful message once that is done. Last but not least, we are going to have a pipeline or assembly function that calls the previous two functions in order and passes the output from one function to the next. And then we are going to call our pipeline assembly function in the if name equals main block. Everything that we've seen so far is just basic Python. So how are we going to convert it into a prefect flow with tasks? Well, we're going to turn the first two functions into tasks with the task decorator. How are we going to do this? Just by adding that little decorator on top of the function. Easy peasy. We'll do the same thing with the save weather function as well. And of course, for the pipeline function, we are going to add a flow decorator. So this is going to represent our assembly function here. Once we execute that flow, we can see that the standard output of the script when it's running will generate a new flow run and we'll get some additional logs so we can see here that there's some information around the task runs being created so the individual functions that are being called from within that greater flow function that's what's happening here or at least that's what's represented here If you were to click that URL that was provided in the standard output when you executed the pipeline, it'll take you to a flow run page in your Prefect Cloud dashboard. And from there, you'll be able to inspect all of the logs pertaining to that flow runs execution. And you'll also get a lovely diagram giving you a nice visualization of the different functions that were called. during the lifetime of this flow run. So the two nodes that are represented in that graph are the individual tasks, or the Python functions that we decorated as tasks. And you can see that there is an arrow indicating the flow of data from one to the other. And that is entirely based off of the dependencies that we set in this script, or rather the data dependencies. Here are some do's and don'ts to keep in mind when defining tasks. You're gonna want to keep them small and modular. That in the long run will help you when it comes to troubleshooting and if you ever want to change the functionality of a task, it'll be much easier to do that with smaller tasks rather than one massive task that does a bunch of stuff. You can use Prefect tasks as a stand-in replacement for Celery tasks as well, and that's because tasks can run outside of flows and call other tasks. As a side note, Prefect is super Pythonic, so don't be afraid of leveraging conditionals wherever necessary. Now we're going to move on to our next concept, logging. Here's a handy-dandy thing to keep in mind. If you ever want to log a print statement, all you need to do is define log prints equals true or pass in that argument into your flow or task decorator. And once you do, you will be able to see all of the print statements that you make throughout your prefect flow inside of the logs or the standard output that gets communicated back to Prefect Cloud. Otherwise, you won't see it because prefect is inherently a polite orchestration platform. So unless you explicitly specify that you want those print statements included in the logs that get sent to cloud, they will not be visible there. They'll only be visible in your standard output. Now I know what you might be thinking, Bianca, please tell me that I don't have to pass in this log prints argument whenever I want to log a print statement. The answer is no. You don't have to do that a million times over. You can do this by default by setting the environment variable that is in this slide. You can set it in your environment or you can just set it in your prefect profile as well. And these are the CLI commands that you would run in order to do that. Something worth noting is that you can change the logging level of prefect as well. By default, it's set to info. However, you can adjust this and change it to debug. if you would like the way you would go about that is by running the cli command down below another cool feature of logging is that you can create custom logs with get run logger so you can import that from the prefect library and here you can see an example in this flow where we are using get run logger to create some logging messages at different logging levels so the first one you can see is an info level log message and the one with the smiley face right below it is a debug level log message so you'll only see that if your logging level is set to debug in your environment if you were to run that flow with info level logging set as the default in your environment, this is an example of what your logs will look like. So everything that you can see here is info level as indicated in the screenshot. However, if we were to enable debug level logging in the environment, you can see that running the same flow produces a set of logs that are way more verbose but in a good way in the sense that it has a lot of helpful information when it comes to troubleshooting so really it's encouraged to enable debug level logging whenever you want to troubleshoot a script in more detail i wouldn't recommend leaving this on indefinitely because it would kind of be a pain to have to sift through all of these logs just for regular day-to-day use. Moving right along to Prefect Runtime. What is Prefect Runtime? It's a module for runtime context access. Many people use Prefect Runtime for labeling, logs, and other reasons. And the information that's incorporated into the Prefect Runtime includes a lot of data around deployments flow runs and task runs here's an example of using that so you can see we're importing the runtime from the library we're also using that log prints equals true statement that we covered in the previous slides but there's something that we're doing in these print statements we're accessing the runtime context for the flow itself and we're getting the name of the flow run. We're also printing out the flow's corresponding deployment name as well. We're doing the same thing for the task down below too. You can see we're using the runtime context for the task to get its name and we're also accessing the parameters that were passed to the flow run that called this particular task as well. As a result of running that script, you can see that the flow run gets created and those print statements are present in the logs there. So you can see the flow run name is there. The deployment name is actually not there because this particular flow run doesn't have a corresponding deployment. But if it did, the name would be present there instead of none. You can see that the task logs are also showing up here. The task is printing its name, and it's also getting those parameters from the runtime context that were passed to the flow run as well. Now let's talk a little bit about states. So why are states important? Well, they're important to anybody that cares about the state of their workflows. And if you don't necessarily care about the state of your workflows, then why are you looking at an orchestration platform video? I don't know. Anyway, here are some examples of states that come out of the box with Prefect. You can see that there are a bunch of them listed here, and all of these are considered not terminal, so they're non-terminal states. These are transitional states. Essentially, it's expected that the task run or the flow run that has any of these states is eventually going to transition to another state. So running, maybe that turns into completed, or maybe that turns into failed. And then pending turns into running, et cetera, et cetera. On the other side of the coin, we have terminal states. So these are things that task runs and flow runs will end with. So canceled, completed, rolled back, failed, crash, you name it. Moving right along to retries. What are retries good for? well they're excellent tools for guarding against failure they automatically retry a task or a flow and you can easily specify retries in the task or flow decorator you can see some examples of what this looks like down below here's an example of a flow that is using retries so we've gone ahead and in the flow decorator specified that this script will retry four times in the event that an exception is raised I believe this script is located in the 102 folder if you want to give it a try. But essentially, this script is inherently designed to fail on occasion. So there will be instances where you run this script and it will complete successfully. However, there will be other times where that exception will get raised. And then you can actually see retries in action. So in a nutshell, we've instructed Prefect to retry this flow four times. in the event of an exception being thrown whenever a retry happens we can actually see some indication of this in the logs so the flow or the task will enter a state a transitional state of awaiting retry and we will then see the result of the retried script in short order what happens if you don't necessarily want to retry the script or the function right away you can actually specify a delay for automatic retries. So beyond just specifying the retry number, you can specify the delay seconds in the flow or task decorator as well. So here's an example of a task that is retrying with a delay that is an exponential back off. And essentially what that means is anytime a retry is set to happen, the Period of waiting between retries will just get exponentially longer as each retry is completed or attempted. Closing the chapter on retries, let's cover prefect variables. Prefect variables are a nice way to store and reuse non-sensitive data. They're represented as key value pairs stored in the prefect database for easy retrieval and reuse. You can create them in a variety of ways through the UI, through the Python SDK, or through the CLI. They can be any serializable JSON, and they're pretty much a stand-in replacement for basic block types that store regular JSON. If you want to create a variable in the UI, you can navigate over to the variables page, and you can create a new one there. As you can see, here's an example of a few variables that have already been created in the prefect ui you can see the name the value some information on when the value was updated and you can also tag the variables accordingly for easy organization if you wanted to create a new variable this is the form that you will need to fill out so you pass in the name the value and the tags as well now If you want to go ahead and create these variables using the Python SDK, you certainly can. Here is an example of how to do just that. So here we are setting the variable at the top, passing in a name and giving it a value, and then we're also getting the variable down below. So we're retrieving that value and we're using it in the context of our script. Building on the concept of variables, let's talk a little bit about prefect blocks. You can consider blocks as a more advanced version of variables. They're the marriage of configuration and code and are very useful for storing reusable configuration that allows you to connect your Python workflows to external systems. Creating blocks is a relatively easy process. You can do so directly in the Prefect UI by navigating to your blocks page. And then all you would need to do is choose a block type that you want to create. And there is a variety of different blocks for you to choose from. Here you can see there is a block for storing S3 bucket configuration. There's a secret block for storing sensitive values. There are even blocks for sending notifications and configuring automations, things of that nature. There's just a bunch to pick and choose from. In this example, we've gone ahead and picked a Slack webhook block. What this block does is it stores information about the webhook URL that we're going to leverage to send Slack notifications to a particular Slack channel concerning the status of our workflows. So here is the beginning of the form. Under the hood, there's really no cause for concern or mystery. Block types are essentially just Python classes. Here's an example of the S3 bucket block class. You can see that there's a bunch of attributes associated with this block, such as a bucket name, credentials for accessing the bucket, a particular folder that we can specify if we want to read and write data to and from a particular folder in this bucket. That all gets stored in this block. Keeping in mind all of those attributes that we covered, That's what's represented here in this form. So if I were to go ahead and create an S3 bucket block through the Prefect UI, then all of this information is what gets captured in the attributes of that class that I'd shown in the previous slide. Beyond the UI, you can actually create blocks in Python as well. So if you want to use the Python SDK, you certainly can. Here's an example of creating a secret block. So we're passing in a secret value to our block, and then we are saving that block with a unique name. If we wanted to retrieve that value stored in that block in a different workflow, then we would be able to load in the secret block by its name and then retrieve the value that is stored within that block. At the end of the day, blocks are reusable, they're modular. and they are the combination of configuration and code. Not to mention, they're also nestable, and they're stored in the server database for easy retrieval, and you can create your own block types by creating your own classes if you so choose. Now let's talk about Prefect Integration Libraries. One thing to keep in mind is that you can use Prefect with most any Python library. You don't necessarily need to have an integration package installed, but it sure does make things easy if there is one available to you. So Prefect integrations are pip installable Python packages that add a bunch of convenience. If you install one of these libraries, you may uncover new block types that you can register with your Prefect account. and it may also contain pre-built flows and tasks that you can use inside of your scripts as well. If you want to learn more about the different integration libraries that are at your disposal, I'd recommend checking out this link. You can see all of the different options that are available to you. Here's a few more helpful resources that you can leverage while you're exploring Prefect. I'd recommend you join the Prefect Community Slack. There are a ton of engineers over there asking questions, answering questions. So there's always somebody that you can learn from there. Our engineers are in and out of that space as well. And of course, there's a handy dandy Marvin chat bot that is trained on the docs. So if you're ever in a pinch and you need to ask a quick technical question, I'd recommend getting in touch with the Marvin bot in the community Slack as well. You can find Marvin in the Ask Marvin channel. All you need to do is at him and ask a question and he'll respond pretty quickly with an answer. Another helpful resource to keep in mind is whenever you're using the Prefect CLI, use dash dash help. So start commands with Prefect. If you don't know all that you're capable of doing in the CLI, then add dash dash help into the mix. Running Prefect-help will give you a list of commands that you can pick and choose from and help you build out a command to achieve what you're setting out to do. All right, we've come to the end of Module 102. You've seen how to understand the state of your workflows and guard them against failure. All the different concepts that we've covered here are listed. So we've covered tasks, logging, states, retries, variables, blocks. integration libraries, and a few more resources to help you on your prefect journey. Now we're ready to start the lab. So what you're going to want to do is navigate over to the 102 folder in the GitHub repository, and you're going to leverage the examples to complete the following action items. You're going to use a flow with two tasks that fetches weather data from the OpenMeteo API. You're going to pass data between the tasks, add some retries to the tasks or the flows, maybe even throw in an exception to force a failure just to see those retries in action. You're going to run your flow as a Python script, and that's pretty much it for the basics. But if you breeze through all of those, I'd encourage you to check out these stretch goals. So there's three of them. try logging the name of the flow run using the runtime context that we covered in the previous slides try creating a block in the ui as well and load in the block in the code and use it as far as the easiest block that i could recommend i would say a secret block would be a great place to start okay let's begin the lab portion of our session today so I am going to get started by showing how easy it is to write a prefect flow that has some tasks in it. As you can see here, I have a little bit longer of an example of our weather script. There are a few more functions than the previous example that we covered in module 101, which only had the one function that fetches the weather. In this case, we have three, and I'm going to... turn two of them into tasks and one of them into a flow. The way I can get started with that is import my flow and task from the Prefect library. And just a side note, I already have a Python environment activated here with UV. I have already installed Prefect. All of that is covered in module 101 so if you haven't already done that then you might want to finish that before going on with the rest of the exercise so with that being said let's tackle our first couple of tasks here so this first function will be a task the second one will be a task as well and then the last one will be a flow and If I save this, what we should see is I will now have a new flow run that calls some tasks. There's my new flow run. If I quickly refresh this on the left-hand side, then I should see that new run, MightyMoth, in the UI. And here we have a nice diagram that shows the flow of information from one task to the other. So as you can see here in the flow, we have passed the output of fetch weather to save weather as input. And that is represented here with this little arrow in the diagram. And if I wanted to look at the information pertaining to each of these functions, I can drill in and out by clicking on them. And then this little metadata side panel will change accordingly. And if I wanted to look at the logs produced by each of the tasks, I can see that in the standard output that is sent back to the Prefect Cloud UI as well. Now, to make things a little bit more interesting, there are a few other examples that we can run through. So why don't we give the retry a chance here so this example will occasionally return a failing status code and occasionally it will fail and occasionally it will succeed so we'll see our retries in action in the case that an exception is thrown here if i clear that up and run this example as is retry flow and we should see yep there was the exception that got raised as you can see here the flow entered and awaiting retry state and attempted to run again and we got a 200 response on the second try or maybe this could have been the third try and the flow was able to complete successfully given that safeguard and all we really needed to do was add this extra retries argument to the decorator. We've also added log prints equals true, which we've covered in more detail. This basically just takes any print statements that are made in the script and then surfaces those print statements in the standard out that's sent to Prefect Cloud. All right, next let's take a look at how we can create a secret block. in the SDK or using the SDK. I'll clear all this up. And if I were to go ahead and run this, then what I should see is a new block show up in the blocks library that I have in my dashboard. So this example in particular is creating a secret block. It's setting that value and it's saving the block. The name of the block is going to be secret thing. So this is. using the Python SDK to create this block, and it'll save it in the database that's used for my server. So I'll do Python create secret block. I'll run that. And looks like it ran successfully. What we should see is, oh, in my settings, if I go to my blocks page, I should see a block that says secret thing. There it is. And there is my secret value. Although you cannot actually see the secret value because it is being obscured from us. But if we wanted to go ahead and access the secret value, we could run another example to retrieve the secret block. And as you can see here, we have that secret value, which we were able to retrieve from storage. All right, feel free to explore the rest of the examples that are available to you at your own leisure. And congratulations on completing module 102. I'll see you for the next one.

Welcome to the second module of the Prefect Accelerated Learning (PAL) series!

In this session, we dive deeper into orchestration and observability in Prefect. You'll learn how to monitor and react to the state of your workflows, automatically recover from failures, and persist useful information using Prefect’s powerful orchestration features.

We’ll explore how to log effectively, inspect runtime context, configure retries, and use Prefect’s variables, blocks, and integrations to make your workflows more resilient and production-ready.

📘 Topics Covered:
-Task structure and best practices
-Logging for observability
-Runtime context and run introspection
-Understanding and managing flow/task states
-Automatic retries on failure
-Variables and blocks for storing configuration and metadata
-Integrations with built-in libraries
-Additional helpful resources

🧪 Hands-on Lab:
All lab examples for this series can be found in this GitHub repository: https://github.com/PrefectHQ/pal-2025-v1