AI agents and durable execution with Temporal and Hebbia | Full discussion

Welcome everybody. Thank you so much for joining us. Talia Goldberg, I'm a partner at Bessemer Venture Partners where I help lead a lot of our AI investments and one of the most exciting things for us over the past few months, quarters, years has been just seeing the pace of development and the changing best practices as companies are adopting different AI tools and products, both to improve their own efficiency and workflows internally, but also to build AI native products and to take advantage of the latest and greatest that's out there. And it's changing so fast and the landscape is so dynamic that what's been really rewarding has been hosting these sessions and this research to runtime series, which brings together folks to share best practices. practices, talk through how they built things, what they're doing, and share their learnings more broadly. So on this session, we're super excited to have Lucas and Max from Hebbia and Temporal. I'll turn it over to let them introduce themselves, but I want to let everyone know we have the chat light open. So feel free to jump in with questions. We'll save time at the end for Q&A, but if you have relevant questions or topics as we go through, don't be shy to put them into the Q&A chat box and we'll definitely try to address them. But thank you again. And with that, I'll turn it over to. Lucas, who what I'm super excited about for this session in particular is that we have Lucas and Max and we have the Lucas who's actually using the product that Max is building. And so you get to hear from someone on how they're actually using it, what they're doing, what's working, what isn't working, do some demos. And then on Max, in terms of what he's seeing, why they built this, how they built this platform and what else it may be useful for. So Lucas, I'll turn it over to you and then we'll go to Max for a quick intro and dive right in. Welcome, everyone. My name is Lucas. I'm a software engineer at a company called Hebbia AI, which is focused on kind of building what we call an AI platform for finance. What that means is basically building a lot of agentic systems that try to solve different workflows within the financial services industry. Yeah, and I've been at Hebbia now about three years, joined very early on when we were about a six, seven person company, now grown to about 100 and recently started exploring the use of Temporal and integrating that. and some of our production systems. Yeah, very excited to talk more about that. Awesome. Thanks, Lucas. And Max, if you don't mind giving your brief intro. Yeah, I'm Maxim Fatih. I'm co-founder of the Timporal project. And before it was built at Uber, we started the project nine years ago. It was called Cadence. Then we started the company and we forked it and we created Temporal. Temporal still is an open source project with MIT license. So it's fully featured in Notion and Gens with licensing. And then we also have a company which monetizes that through a SaaS service. Awesome. And Max, maybe just as context on Temporal, do you mind just sharing with folks a little bit, what was it used for at Uber? What happened? What was the genesis story of then deciding to go and start Temporal? And how has it changed? You started Temporal before we had this latest wave of AI agents and obviously today and what we're going to talk about a lot today is about building durable execution agents using Temporal. But your story started even before that paradigm and you've been right place, right time, right offering. So we'd love to just hear a little bit of that genesis. Just to give you some background, I spent eight and a half years at Amazon and I was tech lead for publish subscribe infrastructure of amazon we build our custom storage engine for messaging and it was well before kafka existed later it was adopted by the simple queue service which still runs on the same back end at least i think design the same design and as a tech lead of the practically pops up at amazon my team and i was involved practically in every big project which and amazon was one of the first companies building a large-scale event-driven architecture at the back end And it became pretty clear that event-driven architectures is not the right approach to build large-scale distributed systems because they're good runtime. They have very good runtime characteristics, but they have very bad design time characteristics because every message becomes practically a global variable, which if you change, you don't know who is broken and just a mess of callbacks and mess of services. And we practically realized that we need to do an orchestrator and we have multiple iterations inside of Amazon and they resulted in the publicly released. a simple workflow service. I was tech lead for that. And later, for non-technical reasons, I moved on to Google and co-founder of Temporal, Samar, who was working with me on a simple workflow, moved to Microsoft. And he built a durable task framework there. And now Microsoft has durable functions, which adopted that framework. And then nine years ago, like 10 years ago, we joined Uber. in Seattle office. And we did the same thing. We built a PubSub system and then we built Cadence, which was an orchestrator. And within three years, we ended up with over 100 use cases at Uber running on that. And Team Portal is a very general purpose system and I can spend some time explaining what it does. But the use cases were from infrastructure provisioning and up to the business level workflows. and obviously payments and big data, everything in the middle, because it's practically the new way to build distributed systems. And then when we started to get an external adoption from companies like HashiCorp, which built their cloud around that, the companies like Airbnb, Box, DoorDash, and a lot of other like Netflix, we started the company and six years ago, and we grew a lot. since the estate now 300 people and hundreds of large enterprises users in production our revenue growing very well so we are in pretty good shape and and it all happened before all the agentic stuff because i think right now people are still playing with agents and doing this kind of memory thing but as people start building real agents like real background like durable agents they need durability they need ability to run right tools which take long time not just a few seconds they need guarantees that these tools will execute people realize that agents are just distributed systems in disguise right that like they just call up apis and what a distributed system called these days you orchestrate a bunch of apis and yes some of those apis are very smart they are llms but at the end it's just a distributed system and to pull solve this problem very well and we didn't need to be with the company we just do what we do and people just who know about us start using us to build agents and pretty successful and i think lucas cannot test for that that's great that's really great background and it tees up maybe lucas can you walk us through like the problem that Hedby has set out to solve, how you went about thinking about what to do, what you would have done without Temporal, what you do with Temporal, that maybe build versus buy decision. Walk us through that a bit. Totally. I think I should probably start with what we actually do because leading with AI platform for finance is very generic. More concretely, we sell to basically a range of clients ranging from investment management firms, ranging across the breadth of alternative asset investment, private equity. hedge funds, private credit. So we sell to a lot of these kind of investment firms and then also to clients on the investment banking side. So that's corporate finance, M&A, stuff like that. And so what do they want? They want a tool that can help them with some of their kind of most painful workflows, where there is a huge amount of unstructured data that has to be sifted through to arrive at insight. So that means like doing diligence, doing kind of market analysis, all these are different flows that may require lots of different information sources if you're diligent in a specific company you're probably going to be diving deep into a bunch of confidential files that are in a data room somewhere if you're doing market analysis you might be doing this very broad analysis and then there's some there's a quantitative component you might want to pull in actual quantitative values to do i don't know like a cash flow analysis something like that so these are quite complex multi-step workflows and the desired output from these or a spreadsheet it could even be something as straightforward as like a simple chat response the first thing i want to make clear is the kind of agentic workflows we're trying to solve can be quite open-ended they can be something that is very quick and something that's very lengthy and a final point on this is that one thing we've seen is actually a lot of our usage is often quite simple. Like often we may build a very complex agentic system and then use a lot of variability that we have to account for. So building these agentic systems, the first way that kind of Max hinted at there is manage context and memory and run things naively as treating it as like a series of API calls. And one of the problems that we run into very early is agents are very slow. One of the first things we encountered was when you're running an agentic system, you can expect a single LLM call to potentially take. in the order of minutes worst case and that's through a combination of issues right one issue is limited capacity you're going to hit rate limits there's a very long inference time that's getting worse now as we move towards more and more reasoning models and then so that's one problem that that we've very early encountered building these agents the second one then is they can be very failure prone that can be a lot of early on especially there were a lot of issues with like widespread api outages and even today we often see one-off errors new models will drop and you'll see a kind of new failure cases. So handling those failures, getting durability is very important. And then related to that, one of the problems we get with LLMs is stochasticity. You can imagine LLMs themselves like somewhat random in the output. You chain them together, create an agent, and then you end up with multiplying that stochasticity. And we find we actually can't really solve for this. The best thing we can do is at least trace those outputs. And so all these different issues combined with kind of need for flexibility and extensibility as I described earlier because the use cases can be very changing meant that we really need to rethink like an actual platform for running this and as Max was hinting at I think it became clear to us this is not just a series of chained API goals this is actually a real distributed systems problem especially when you consider running longer term deep research agents as we call them that have access to many tools that might be web search or might actually be the tools themselves might be very long running, right? The tools themselves can involve like more complex steps, like searching through a private folder, re-ranking the results from that folder and so on. So these are all the different challenges that we encountered. And I think that kind of leads directly into what, what Max was mentioning, which is temporal can be a way to address all those individual challenges. And yeah, and actually a final layer on that, which I think also could be an interesting point of discussion is that there are some out of the But there are some ways right now for that. For example, OpenAI has a responses API that kind of makes it a lot easier to manage context and do things like that. One issue you will find when you're applying these genetic systems production is your customers are going to be terrified about security. There's a wide ranging fear about feeding data to OpenAI or Anthropic and definitely not XAI and having these companies training on your proprietary data. A lot of these kind of solutions that are now being presented by foundational model companies, OpenAI, etc., are off limits for now in some of these domains. So you actually have to do a lot more of the engineering yourself, we found. You have to actually figure out how to manage that context yourself. And because you cannot rely, or at least from the perspective of your customers, you don't want to rely on a company like OpenAI to handle that. And so because of that limitation, it becomes even more important to... to have a really robust system for tracking the context and state of an agent system as well. And that also comes in there. And Lucas, before you implemented temporal, were there certain code patterns? Maybe it's retries, maybe it's exceptions and handlers that were littered throughout the code base where you were like, gosh, you just said at one point, gosh, we got to go. We got to go look for an external solution. Yeah. So you mentioned retries. Actually, that was something I would say retries was something that we had not handled well until, simply put, until we started adopting Temporal. I think we more adopted an approach of trying to preempt LLM failures. So we built a very sophisticated kind of distributed rate limiting system that I think is still a key part of our stack. But that was more based on the premise of, okay, can we preempt errors that might happen from overloading a given API? But we didn't have a great retry policy. Part of having great retry policy is actually being able to have really accurate tracking of the internal state of your agent system. So you can say you can checkpoint at a given point and then restart from that checkpoint, which is actually not trivial to implement. And so I think that's a great example of where temporal accelerates things. On that note, with the rest of retries and deep research, maybe we jump to the demo. I know that's something we're all really excited to see. And we can use that as a jumping off point to then discuss. some of the design decisions for Temporal and other questions around building deep research agents. Perfect. That sounds good. I'm going to start by sharing my screen here. Awesome. So, first things first. Does everyone see my screen here? We do. Okay, awesome. So actually, I'm going to start my demo. So I'll give you a little bit of background on it. Essentially, I was told I couldn't demo Hebbia's production system, so I had to build some kind of demo here. turns out that max and his team have actually done a lot of the work of kind of building some very robust samples how to use temporal so i wanted to start actually by pointing people to this this repo i can send a link to which temporal has built which actually provides some pretty good starting points for anyone interested in this so i'm actually gonna the way i'm gonna frame this is i'm gonna start by walking through an existing example that the temporal team built out and then show how i might extend this to recreate some of the the kind of longer running deep research workflows that we want to do for an action like heavier i'm going to start actually by walking through what one of these workflows looks like let me close the right okay cool so do you guys see my code we do yeah so what we have here basically and i'm gonna this is actually a financial research agent that has been implemented by the temporal team let me make this bigger which was my starting starting off point and essentially what does this consist of at its core we have a worker right the worker is going to actually be responsible for the execution of this temporal workflow and you'll actually notice that temporal actually has a pretty neat feature here so ordinarily when you write a temporal workflow there are this concept of activities right so the workflow is might be i don't know run deep research within that workflow there may be actually many different activities that have to be executed and max can elaborate i'm sure at some point on this the structure and how that kind of works behind the scenes but temporal neatly implemented at this kind of these kind of plugins now that actually allow you to kind of abstract away some of the definition of what of what is an activity and what is not and this plugin here is the openai agents plugin what this allows you to do is it basically will wrap any llm call that's made through the openai agents api and treat that as an activity and that kind of very neatly And as you'll see, when I run through the code, it very neatly allows you to implement kind of an agent system without actually thinking too much about the temporal division of labor that's happening behind the scenes. But yeah, so going into this worker, we basically define here a workflow. And here the workflow is going to be a financial research workflow. No surprise here. And it's going to pull, it's going to pull tasks off a task queue. And temporal, I believe like these queues are dynamically allocated. You can, it's quite easy. from a user perspective to spin up different queues, spin up different workflows. And if I click into this specific workflow, this is going to be a really simple one. We have this class that they've implemented, Financial Research Manager, and we're just going to run it. And I'm not going to go into too depth on all the code here, but we can always follow up on this. But essentially what they've done here is they have a series of steps that are being executed one by one. And in the run method here, let me make this bigger. We begin by planning the searches we need to do to resolve the user's query. So the user's query here might be, I don't know, write a report on Apple's performance last quarter. So we go to the plan stage and the plan stage is running this planner agent and go to the planner agent. It's being defined here. And this planner agent is literally just an LLM call with O3 mini with this prompt. And the prompt here is saying, basically, you are a financial research planner. This stage of, this is the first stage of our agent. It's basically just decomposing the user's request. into a number of like individual search terms, which is a pretty standard thing you would do in agent workflow. You have a complex query, you want to split it into subtasks. And then if I go back to the manager here, we run this, we plan the searches, we then run those searches. I know I can go into too much detail here. This is just doing basically an opening AI call with web search as a tool. So the LLM can use the web to finances. And then we have a final stage where we're writing a report, zipping together all those search results into a final cohesive answer. And there's a verification layer where we can basically check the output against sources, make sure that there's nothing that looks fishy. So in a sec, just to demonstrate what's going on and how temporal works. And then we'll build from there and I'll show you an improved version of this. Let me show you what this looks like. You should now see my terminal. I've already got the worker running. So for this demo, you can imagine the temporal worker running. And now I have a script that's going to basically enqueue this workflow. And the way that works is if I go back here, here is the script. It's going to ask for query. And then it's going to spin up a temporal client and then call client execute workflow. What this is doing is it's basically just enqueuing a financial research workflow with a random like ID, putting that on the task queue and it'll get picked up by the worker. If you don't mind me asking, just seeing the code pattern, it reminds me a little bit of the lane chains, the LOM indexes, the Haystacks, the traditional LLM orchestrators. Of course, Temporal has come from a different direction, is more of an emphasis on production readiness and resilience. But did you guys ever think about using other tools such as some of those traditional LLM orchestration products? And maybe Maxime, after he answers to that, I'm curious to hear your perspective on where Temporal sits in the market. yeah i'm sure as maxime will point out much more effectively than me but what temporal is doing is fundamentally quite different it's actually completely different as i would say to what those tools are doing um this example being a very toy example where everything is running on my laptop and is not a great example for that but because we actually have a a cloud hosted queue we have the ability to actually run and a temporal server right that can be hosted on temporal cloud or hosted self-hosted you this is actually a true distributed system, right? And while it's not, while the worker is also running on my laptop here, the worker could be arbitrary. The worker could be running in an isolated manner, which is how it would look in production system, which is not something that's achievable with. In other words, it's not something achievable as Langchain or these orchestration libraries. In other words, Temporal is solving kind of an infrastructure problem rather than simply a kind of code cleanliness problem. I think it can be, can you just show again the workflow run method just to? Yes. a little bit explain what's going on there. I think one thing, yeah, no, just financial research manager, Maran. I think one thing to look at that, look, it's just normal Python code, right? This is a normal Python, I think, IO code. So you look at that and say, okay, what's interesting about that? When you run it in the context of temporal workflow, the whole execution of this code is fully durable all the time. So the full state of these executions and including the agentic loop itself, right? Because when you say agent run, you go into the OpenAI SDK library, which just contains a loop, right? And every step of that loop is durable. So what it means is that if your process crashes any time or something happens, or you block here for any long time for them waiting for user output for even for days, you don't need to do anything, right? The moment that you need to do the next step, this code will be resurrected in exactly the same form with all the variables, all the fields, all the like blocking calls still blocked on that call. and it will continue executing as nothing happened. We call it durable execution, right? So I think the power of temporal is not even that it's like scales and it's fully distributed. It is that you write normal Python or like we have eight SDKs or nine SDKs now in your language and this code is fully durable. That's why it was used before agents because if you need to do money transaction or transfer you need to make sure that this completes. We guarantee completion of code in presence of failures. So this function can run for a year, right? If your report takes one year to complete, it doesn't matter, right? Because it still will be blocked on this line of code and wait there. And the moment it needs to proceed, it will recover and continue executing. So I think this is the power of this, what we call durable execution concept. It's a new concept that didn't exist before when you actually write normal code and this code is fully durable. And this is the biggest difference from other approaches like graph based approaches, because why do you use graph? Use graph to make it durable, right? Because you want to make sure that because you cannot write normal code. you try to invent something right like all these traditional workflow engines they do that right they use yaml they use gson they would do instead of just normal python code they have to and complexity workflows is not in the steps right like in the its complexity is in the actual data passing and can you open the perform searches method yeah so look this thing is actually running these tasks in parallel right we do i think you create task right and then we just wait this task to complete So here we just did practically split join. It's just as normal Python code, but this is highly distributed thing, right? Like we're running these things in parallel. If anything crashes in the middle, you will recover. You don't need to redo those steps. And I think this is what's important. Temporal workflows are just code, but they are workflows because they are fully durable. And I think this is what we need to understand from this. That's why it's so simple to write here. Try to model that any other way, right? If you look like graph-based implementation, they cannot do dynamic stuff because the graph is not dynamic. So practically, you'll need to hack it by creating nodes. to run things in parallel, right? Or you need to do some other things. So you need to do like a JSON structures manipulation and adjust normal Python code. And I think this is the main power of Team Portal. It's just better developer experience. And to demonstrate that, maybe we should just run this and show what it looks like. And you can kill the worker in the middle just to demonstrate what will happen. All right, let's try it. Let's do a complete run through first just to show what it looks like. Here, I'm just running that workflow script I mentioned. I can enter a query. I can say... Right. Let me make this bigger. Write a report on Apple's most recent quarter. And now it's going to run. There's a convenient UI here I can go to go and refresh this here. I can see the actual running of my workflow. And you'll see here, this is, I think, a good demonstration of what Max was talking about. We were actually tracking in this event history, every single individual activity and activity here being an LLM call. And I can look and I can see the actual inputs and outputs. So here, for example, this was a web search we ran. I can see the system prompt that the LLM was given. And I can see the model settings. I can see all this information and get this like nice water flow view of everything that is happening as we are running this workflow. And just to demonstrate failure, if I kill the workflow or kill the worker here. Oh, it actually completed before I killed it. I'll try that one again. But here I'm just running the worker again. And let's resubmit the workflow. And you can run as many workflows in parallel as you want, right? It can have billions of them running in parallel. And you don't need to change anything besides having enough workers to process the load. So now we have a new workflow. And now we'll make sure this is running. It looks like it is. Let it complete at least one activity. Yeah, yeah. All right, here we go. So we finished the first stage, which was producing the web searches. And now we interrupt now and we can see here it's failing and it's going to retry. You see worker shutting down. This activity did not complete in time. Another common failure point you might see here is, for example, LLM kind of timeout. You can set timeouts in individual activities. I think here we have maybe like a one minute timeout. If the LLM took longer than that, same behavior would happen. Workflow gets canceled and retried. So let me spin the worker back up and hopefully it should get resumed. there we go so now it actually retried and completed and i can see here i think the number of retries let me make this bigger look activity task started this one here right yeah yeah click on it and it will show the number of retries this one here right yeah no it's attempt one you probably want to attempt one yeah yeah you can see in the visual show yeah yeah this one was retried two times and it shows the last error yeah oh yeah Maxime, can you speak a bit to how is the temporal service capturing all the variables and all the state of the execution? Is that through the worker? So I think one thing is important to understand is that execution model, especially for banks, as they always ask, how do we use cloud? So the execution model is that you run code locally. Temporal is just an SDK, right? Like for them, in this case, is Python SDK. And this code is part of your application. And then before sending any information to the backend cluster, and again, you can run an open source cluster yourself, it's fully featured, or you can just point to Temporal Cloud, which just runs the backend cluster, you encrypt everything. So all the information is fully encrypted and stored there. And then when you need to take next action, this information is sent in encrypted form to your worker, and then you can decrypt it using your keys and algorithms. So that's why it's super secure, because we never looked at the information. And the way it does it, it just records the inputs and outputs of all events. So it uses event sourcing internally. So if you have this history you were looking at, right, like you were just showing the history like this, like underneath these events, like at the bottom of the screen. Yeah, I'm just closing this out. Yeah, so at the bottom of the screen. So these are events, and we record these events, and we replay the code from the beginning. And code should be deterministic, but... Otherwise, it can be fully dynamic, but it's deterministic. And we recover state by replaying the code. That is the basic idea. And it gives us huge visibility because you can see everything which happens. So all the steps are always recorded, all failures and everything in the middle. And then the other approach benefit, you can actually download this history as a JSON file and replay in the debugger. So if you get a production issue, you can just download the history and replay the debugger as many times as you want. So it's practically every time you see any production issue, it's 100 times easier to troubleshoot than any other system when things are just a bunch of files and correlations and traces and so on. We also integrate this metrics and tracing if necessary, but you actually can open this workflow in the OpenAI tracing and you will see the traces there. Yeah. And in our case, the heavier because open AI tracing is not compatible if you are doing a zero day retention policy, which is quite important for security with a lot of these kind of enterprise AI applications. We actually cannot rely on that. So temporal or alternative tracing providers are only source of truth if we want to debug something like this. And you see here in the sample, all the data in the UI, but in the practical, in real scenario, they will be always encrypted and then you decrypt it in hitting your internal decryption service. yes and i believe actually there's an option as yeah when you download right yeah this makes that clear right you can download and the download option mentions codex over here what this means is you can host your own server to actually do the encryption and decryption so you have ownership over that security We had one question come up in the chat that at some point, if you don't mind answering, I think would tie in nicely, which is someone has just started looking at Temporal a few weeks ago and is curious how heavy it scales its database for the history service specifically and what you use and what your strategy is there. And that might tie in nicely at the end to just talking about how you think about this demo and maybe you too, Max, like taking a demo and then actually going into production and what the differences are and what's needed. Yeah. the question pointed out correctly you're going to need choice of database to back all this up to back up all this event in history use postgres for that it's pretty easy support for that with temporal and then your typical kind of calculations involved there of like how you scale that instance i don't know how much depth you're going on that but you can see event history is actually quite it's going to be stored in an encrypted state right it's yeah it's not going to be one thing to consider that is how you handle encryption decryption but on the database side itself i don't think there's too much complexity and if you really need a really scalable solution you can use cassandra and cassandra you can scale out to pretty large cassandra clusters and if you don't want to do it yourself just use temporal cloud you can sign up with your amazon like amazon and google credits and we give promotions and we give startups get credits and the reality is that using temporal cloud is cheaper than running yourself at least at a reasonable scale. And if you're concerned about security, as I said, we have very good security answers. We have largest banks in the world using temporal cloud. So talk to us. We absolutely can explain you why you actually can pass your security review. and still use our cloud solution. And our cloud solution has options for 4.9 availability and is used by hundreds of companies. Every coin-based transaction, they use open source, but things like Snapchat story, every Snapchat story uses our cloud, for example. And there are a lot of using banking as well and financial services. And even before agents. Again, it's not only agents. You can practically build any solutions which require durability. Maxim, we've got a question in the chat around, how Temporal handles extreme scale, say millions of workflows a day while ensuring zero data loss. And maybe we can use that to talk also about what were some of the big challenges in building Temporal and what are challenges that the company is actively trying to solve today? So first, millions of workflows a day is not large scale. We can do hundreds of thousands or even millions per second if necessary. As I said, every real Snapchat story goes through our backend and during New Year's Eve you get a lot of those. So millions of workflows a day is a small scale and it will be pretty cheap. I think we charge on the highest price without any discounts is $58 for a million actions. So millions of workflows will cost you peanuts. And how we do that? It's 20 years in the making and this code base is nine years in high scale production, starting from Uber and used by thousands of companies at large scale. And it's just a very hard work of hundreds of engineers and like over time and right architecture from the beginning. And again, right architecture because not we are so smart because we did it fifth time because and starting from Amazon because even publicly released simple workflow service was the third release. we had two internal releases and then we did it at the tuber we did it at microsoft so we just spent a long time thinking about these problems there is presentation called designing workflow engine from first principles If you Google my last name and first name and design and workflow engine for first principle, you can find the link. It was a meta at scale, like Facebook at scale back then conference, and I presented that and explains the kind of backend architecture of Temporal. It's share nothing, like scale out. We never hit, so far we weren't able to find the limit of the platform. It's always a saturated database. You can give it any database and it will scale out. For our cloud, we built our own database. We practically create a database just for that use case, and that's why it can scale almost infinitely. So, yeah, scale is not a problem. Availability is not a problem, especially if you use our cloud. Perfect. Any more questions? Maybe one more question from the chat. Are there dark patterns or common failure cases, maybe both for Lucas and Maxime, that you see commonly in the industry when people are trying to... implement either temporal or some sort of other durable execution workflow i can start with an obvious gotcha and then probably lead into maxine will probably have more detailed answers but one obvious one when building agents in particular is there is a sort of determinism constraint on the workflow itself what this means is that you cannot have let me show you an example you cannot have non-deterministic code execute on the workflow itself this now on paper this seems like a big problem right because llm's non-deterministic but what this means is you can still have deterministic code run on an activity level and if you're using the agents openai agents plugin that i was just showing um you never have to think about this right because this will isolate each llm call to an individual activity but one gotcha you that you have to watch out for is you can't just be throwing llm calls into your workflow you like if I'm not using this agents plugin, I have to be thoughtful about how I am running LLM calls. An example of that, maybe if I, we don't have time to do a full run through of this, but the kind of follow on that I just quickly build up for this is a version of deep research where we are doing this like iterative exploration of subtopics. I couldn't, for example, here say, let me, I don't know, randomly choose a set of subtopics to explore at a given time on the workflow level. That would not be permitted within Temporal. that that's one thing to look out for. I think one important point is that people confuse determinism with dynamic execution. Workflows can be absolutely dynamic. They can, as you see, they can run, you can run random stuff, you can do dynamic stuff. The only difference is that if you use random, you used to use specific API we provide instead of the system random. Because on replay, we will provide the same thing. If you use time, you can use our API for time. But otherwise, workflows are fully dynamic. And you can absolutely write code which like calls LLM, say give me a set of tools, and then this set of tools is dynamic. You can implement any dynamic requirements. You can even, for example, very common use cases like implementing domain-specific languages. You can give a JSON file with your whatever domain-specific language, and you can write an interpreter as a workflow. So it's very common for us to interpret existing workflow definition languages like in other domains. And when we say determinism, all it means is that don't call external APIs from workflow directly, call them through activities. But this is practically the main limitation. And the practical limitation is that activity results should be serializable. We usually use pedantic in Python for serialization. So if your input outputs are pedantic serializable classes, then it works just out of the box. And I just see there's another question here on how do temporal and or heavier detect hallucinations for that many workflows. I'm assuming you're talking specifically about LLM hallucinations here. So that's, I'd say like a separate challenge. I'd say that is in the case of heavier, that is all to do with how you do the prompting and orchestration of agents. What I mean specifically here is like separating tasks. Hallucinations are made more probable the more you ask an LLM to do. And we, the kind of approach we take is getting LLMs first to just return extracts or quotes. And by making decomposing complex tasks into very small sub problems that can be verified like the quote extraction for example you can verify against original document you can mitigate the potential for hallucinations any other questions i don't i think we've answered most of them i see the question about about link chain right link smith yeah can you read it So someone asked, can you speak a little bit to how this setup is different from what you can achieve with LangGraph as an orchestrator using LangSmith for tracing each agent execution? Any pros, cons would be helpful. Maybe let's start with Max. And then I think that ties into Lucas, like you went with temporal. What else did you consider? Yeah. Maybe I'll start. Max, why don't you help us speak through the setup? So a couple of things. First, you can absolutely use LangSmith and LangChain temporal. You can absolutely run LangChain and activity like nodes and then you can link them through temporal workflows. And we have a sample. In this repo, there is a sample doing that. And it integrates with LangSmith. And we even integrate at the total level, you'll see even activity invocations, workflow invocations, child workflow invocations. There is nothing special about that. The LangGraph is a little bit different because so LangChain is pretty kind of popular library and it does all sorts of integrations. So I think it's absolutely nice to use that. And LangGraph is a... Also reasonable, a length myth is a very reasonable solution for tracing. But a LANGraph, a problem with LANGraph is practically an attempt to do a workflow engine, right? Because the reason LANGraph exists, people want to persist state. And just, I actually wrote a blog post about that. Graphs is just the wrong way to do that because technically, instead of writing normal code, as you see here, you practically try to match, put this code as a graph. And this code is highly dynamic. Graphs are not dynamic. And that's why they need to cheat. For example, in the case of temporal, the whole agentic loop is durable. right? So you call LLM and then you call tools and then tool can be long running tool which takes three days, right? And then tool returns in three days and your loop goes to the next iteration. You go to LangChain, LangGraph, it's actually the whole loop is just one node, right? So they practically cannot use graph for that. They practically just go to the Python code but they cannot checkpoint state in this case, right? And the same thing if you look at the research sample from LangGraph, you will see that they have node which runs parallel searches. Why? Because graph cannot run parallel searches, right? Graph is not good for dynamic stuff. And then you think about data, right? Like, okay, every time you need to mutate data from the graph, it's a mess, right? Because graph practically has a bunch of global variables or structures because it's not clear, right? Here it's just normal method. Method gets input and output, right? You have an activity input and output. So it's pretty clear what's going on and it just Python code. Error handling is just do try catch, right? And you can practically write normal, like finally. your error handling logic is trivial. But if you have graph, again, you're practically trying to kind of get your normal code to comply to the graph architecture, and it will be a problem. And then there are problems about runtime. And again, I didn't run against the runtime myself, so I don't want to say it's bad, but reality is that for an open source solution of Langraph, it just doesn't support recovery at all. Because Langraph, all it does is checkpointing. But if your process crashes in the middle, Something else need to tell, recover this thing. And they don't have that in the open source, right? I know that they have some sort of like queue, task queue on the backend in the, but I don't know how well it works. Does it support durable timers? Does it support all the types of timeouts? So I think if you're like for production, high scale use, try it yourself. But I have my big doubts about line graph being able to execute on that. But I think the main thing is not even like reliability and scalability, which I think we are certainly the best in the field. in this area, but it's mostly about programming model or graphs and just run way to write workflows. And I think I can take the second part of that question, which I think was asking specifically why have you chose temporal? And I think the reason is our emphasis here was on solving for the scalable and durability that the Max was just talking about. I think actually the summit, like the layer of agent orchestration, if you want to call it that the layer between the LLM call and everything else that wrapper is quite easy to build yourself. I want to say like we have ended up early on using some external tools and then made a decision to kind of implement our own agent framework for orchestration just because we had some custom requirements around doing dynamic routing between different models, unique kind of rate limiting system that made it difficult to use an out of the box solution. And that part is, I would say is not, it's something that is doable if you want to implement your own. The difficult part. as Max was getting at, is making that system durable. And that is what we outsource to Temporal. And because Temporal, I think, has a very strong case for solving that specific problem very well. I think just reiterate that Temporal doesn't have its own framework because we integrate with any framework. Again, we can run link chain. We can open the SDK. You saw the sample using open SDK. Pedantic AI integrated with Temporal as well. So you can run a pedantic AI workflows. And in a lot of cases, you just don't need any of that. You can just write. A lot of people say we don't want to use framework. We just want to write code. But if they write code, this code is not durable. But if you write this code as Temporal workflow. the code becomes durable so you can have your own event loop, you can have your own agentic loop, you can have your own flows, you can have your own framework and Temporal would support that. Our kind of strategy is not trying to create yet another framework. Our strategy is to be use what's the best for you and we will provide durability and scale and availability for you and be an underlying engine for all of those. that is i think the most important kind of point there is you are not linked like you practically don't need to be tied to any specific genetic framework if you use temporal and you can build your own and the other big part which we didn't have time to mention here but i think worth mentioning temporal has very strong framework about inter-process communication right like we have this project called nexus when we practically i'll think about rpc and service mesh but requests are guaranteed to be delivered and they can be long-running So you can make an RPC call and takes three days and returns later, and you can practically block on that and wait for that. And now you can have long running tools, right? Now you can have agent to agent communication when agent takes long time to complete work. And you can have, so I think it completely changes how you think about your system. It's not a bunch of events, right? Because again, events, LLMs don't understand events, but LLMs understand tools. And having the request reply semantic for long running asynchronous operations is very powerful. And it works at any scale, right? As I said, you can have billions of those workflows running in parallel. That's awesome. We've just got even a couple more questions. I'm loving the questions because I think they're really relevant broadly to everyone hearing the answers. And so maybe we can just go through these other two that just came in. Let's start with, can you have a hybrid approach? Someone's using Google Cloud and has their own servers. Can you orchestrate everything in sync with Temporal? How does that work? Yeah, absolutely. The main thing is Temporal. Again, Temporal is just a library and it's part of your application. So you can deploy it anywhere. We have even case studies when people had these orchestrating workflows across multiple regions. We have multi-region orchestration and this RPC layer, which I said we call Nexus, works across cloud providers. It works across regions. And you can even do with Temporal, you can do things like migrating your workload from one cloud provider to another without downtime. So if running on Amazon, you can move your workload to, for example, GCP without downtime. And we have features for that. Or you have open source cluster, you can move to cloud without downtime. We have features for that. And we are working on features to move back. You're now cloud, you can move back to OSS, and we guarantee compatibility between OSS and the cloud solution. So I see other question about, can you use your other database like MongoDB and Postgres? Postgres is officially supported, so you can run it yourself. MongoDB, I knew some companies did MongoDB binding, but I don't think it was to the quality we would support. So yeah, we officially support right now MySQL, Postgres, Cassandra, and SQLite. And then our cloud has its own database, which is proprietary. Great. Maybe one last question. For you all, anything that you think is still very unsolved in this LLM orchestration ecosystem and where you're focusing your time, both Lucas in terms of your development focus and temporal in terms of where your product is focused? Yeah, sure. I can give my perspective there first. One thing that I think many of the existing frameworks handle poorly is like a deep nesting of sub-agents in these agentic systems. I think it's... It can become quite tricky to manage when you, so we, for context, we, the examples I showed and the example, I was just starting screen share at the end, either work on a very fixed kind of ordering of agents or have a kind of orchestrated sub-agent hierarchy where there's one LLM responsible for managing other LLMs within it. But in reality, As you move to more complex production systems, you very easily end up with a case where you have an orchestrator, suborchestrator, and then LMs within that, and you can have deep nesting of these agents. And that's something that I think can get quite difficult to manage with any framework that's out there right now. So I would say that, in my opinion, from what I've seen, is not a fully solved problem. And then the other thing that we've been thinking about is prioritization. This is something that Temporal has actually put up a... a feature for they have a task queue priority feature that i think is in beta but yeah the point of this is basically as you move to longer and longer running tasks being able to dynamically prioritize and deprioritize a given task becomes very valuable given you're generally dealing with finite capacity right you only have so many tokens you can move through in a minute through a given llm api right it's very valuable to say okay this is a background task i'd want to deprioritize that or you know well, this is a high urgent one that I want to reprioritize. And from what I've seen, I guess this is also a question for you, Max, is currently the temporal priority queues, they are kind of a static priority that's set when you enqueue. It would be really cool to have an ability to kind of reprioritize and have, I guess, like an efficient priority queue implementation where we could... update priority for a given workflow dynamically. So that's another thing we've been thinking about. Priority is per activity and vacation. So you can change it anytime. So practically every time you invoke next activity, you can change the priority of that. Or like you've run two activities in parallel, one of them have one priority, another can have another priority. Okay, so not just on the workflow level. It's on that. Workflow is just a high level, right? Because you just, right now we only do priorities on activity level because workflow is on... don't execute them for a long time. Execution itself, they are mostly blocked, waiting for activities. And activity invocation, each of them can be separate. We also added fair queuing, which means that if you have a large number of your clients, right, and tenants, we can provide fairness across them. So talk to us about that. Perfect. One thing which I think is not solved right now very well is tools, long-running tools, and durable execution of tools. MCP doesn't have good support for long-running operations, right? MCP was kind of conceived for use cases when every tool invocation takes a few seconds. But as soon as you have tool, which requires some kind of some time, right. And need to grant execution. MCP doesn't support that. We kind of voiced, I think there are people talking to the committees about that. We are part of that committee, so it should be fixed. But in the meantime, you absolutely can use temporal to invoke tools without, and we will integrate with MCP and provide capabilities to for long running tools. So if you have a need for your agents to actually call something, which is long running. Talk to us because I think there is no good industry solution out there right now. It is standard solution to do long running tools and the execution. There is a question about ORM about like Django. You don't need a database in Poro because it's durable execution. Every variable is durable. So practically just like in that code, like every query, everything in that class is always durable. So you really don't need database to implement the whole thing. It's already durable by default. So this is exactly the magic of durable execution is that every local variable is already in the database for you, right? And if it stays there as long as workflow is running, and even after that, because you can still query it. So if you want to update database, you can always do it from activities and use whatever framework you want to run limitations. But in a lot of use cases, a new database at all, because temporal just already persists state for you. Well, team, I know we're about time. This was awesome. I loved all the questions that came through the chat because, oh. We had one question on running the AI generated code. Did we forget to do that in the last demo? Do you guys have one last second? Lucas, is it still up? Oh, I, uh, let's see. I've closed my screen. We may have one more second to complete the demo. We can try and do that. Otherwise we'll give everyone their time back. Yeah. Let me run that right now. By the way, temporal is very common target of AI generated code, because if you think about white coding workflows. It's AI is not very good generating like very scalable applications, but a web coding temporal workflow is actually not that hard. And thanks to everyone for all the great questions. It was awesome just to be able to make this as practical and useful for the audience as possible. Okay. Let me share my screen real quick. Okay, cool. So here we have a vibe coded deep research agent, and then we go to, to run this. Let's do a, let's do a same query here, like a write a report on Apple's most important. and now we are running this workflow and we can track it see what's going on here so here we should see this kind of these two stages right where we are exploring different subtopics and exploiting them and it'll probably be much longer running workflow this should take probably about four minutes but it should look a little bit more interesting so here we go here are the sub queries oh this is cool so an interesting point here is you know this could but for a fixed set of steps because any on this and see my zoom. No, this is great. Cool. Thank you. We really appreciate it. If you have other questions about Temporal or looking at becoming a customer also feel free to follow up with us and we can figure out the right way to put you in touch. So thank you all so much and thank you to Lucas and thank you to Max in particular for spending your time and being so thoughtful and sharing just really practical insights with the group. Thank you both. Thank you. Bye-bye.

A technical deep-dive featuring Max Fateev, CTO of Temporal, and Lucas Haarmaan, Founding Engineer at Hebbia, as Bessemer's Talia Goldberg and Bhavik Nagda explore how to build production-ready deep-research vertical AI agents using durable execution frameworks. Temporal provides the industry-leading durable execution platform for orchestrating complex, long-running workflows at scale, while Hebbia is pioneering vertical AI agents that conduct deep research and analysis across vast document repositories.

This is part of a broader series, Research to Runtime, focused on entrepreneurs, developers, and technical leaders building with AI. Visit https://researchtoruntime.com/ to sign up for future events.