Durable AI Agents with Restate - Community Meeting July 2025

Yeah, welcome to our community call. I'm Stefan, one of the founders of ReState. And this community call is basically a bit of an interactive demo of some of the examples and integrations we've built recently for agentic use cases. I guess it's a bit of cliche, everyone does these days. It actually turns out Reset is genuinely actually a very nice tool for some of these use cases. And I hope I can actually show you a little bit of, yeah, a few nice bits there. It's also gonna be a bit of just a demo of some of the more recent features in Reset specifically around like UI and observability and so on. So I hope that's gonna be nice too. If you're interested, some of the background of what we're doing in this call is also in the recent blog post that we wrote. If you've got our blog, non-reset.dev slash blog, durable AI loops. Some bits are also here. We're going to go into a bit more detail, more visualization, a bit more explanation here. I think we need to do a few instances of this one because initially as we were setting this up, we were thinking we can actually show a bit how we can use Restate across a lot of different AI use cases and SDKs and different languages. But it turns out, I think we're not gonna get through all of those within an hour. So let's actually start with one. Today, we're going to do TypeScript and the Vercel AI SDK, and I think we just have to do another one in a week or two. And use, for example, the OpenAI SDK or the Google AI SDK. And that allows us to show us a few different patterns, because all of these SDKs kind of have different philosophies and takes, and it allows us to show a little bit how. you know the whole philosophy of how reset works with durable execution and you know state and durable communication now this is a really nice orthogonal piece um across different sdk so it's not a it's not a specific sort of agent runtime for for any specific solution but really the mission is sort of more general purpose um resilient applications and um It looks like we have a few familiar faces, but also a few names that don't recognize, like folks that are here maybe for the first time. So let me take maybe one or two minutes just to give you like a very brief introduction into Restate and generally how to think about building reliable applications with Restate before we go into some of the details, okay? We can actually do this introduction with a small agent example as well. And yeah, I'm just going to go into code right away and share my screen. And by the way, I'm going to do a pause every once in a while and collect questions. Feel free to collect them in the chat in the meantime. And, you know, like raise your hands if you have questions and then like every... you know every couple of minutes we're going to do a pause and then go through all the raised hands and the questions okay all right let's do that good so uh here we go fingers crossed this is uh this is very much like uh alive and everything so um let's uh let's see if the the demo gods are actually um actually good with us today okay so The simplest way to think of a restate application, independent of whether you're building an agent or not, is that you're basically turning something that is like non-RPC style application with request handlers into a durable application. So what I've written here is actually something, it's completely without restate at the moment. It's just a simple mini AI application. that runs as an express JS application. You know, it runs on port 8080. It has a single route, agent run, takes a request, just forwards that to our function here, which itself is actually very simple. It just takes this prompt, sets up a model and then uses this function. And this is all basically just Vercel's AI SDK. And for those who've never used this before, it's a fairly lightweight SDK that that basically is a tool to help with interacting with, you know, across different providers, like we're using OpenAI here, but you can also use Anthropic, Google. And you can specify these, yeah, these interactions. Here's a system prompt, here's your user prompt. You can define a few tools that you make available. And then this is basically sort of a single, yeah. a single interaction, which in itself can have a couple of steps, for example, if it involves tool calls. So if you do multiple steps, then each individual step will be an interaction with a model, which might tell you, okay, you're actually done. I can directly give you the answer or call a few tools and then tell me what the tools say. And then we're gonna do another round up to a maximum of five. So, so far, there's nothing restate specific about this. It's really just this simple function, the simple handler. and we can um can just just simply start this let me see that i don't have anything else running on that port no it should be good that should just bring this up this doesn't actually look right no it's just the log output that's wrong it's just the log output that's wrong it's actually listening on 8080 okay anyways um okay and let us you know send this send us a single simple um invocation you know Just going to call local 8080 agent run and the prompt is, you know, what's the weather in Berlin? That's the prompt we selected because the one tool we actually made available to the agent is a tool that allows you to, you know, allows you to access the weather. Okay, it tells me the weather in Berlin is currently sunny with the temperature 23 degrees and, you know, this is all of it so far so good. Let's look at what we would need to do to turn this into a reset application. And it's not a lot. The philosophy of reset is really, we're trying to keep the shape of the application as close as possible to this type of sort of request RPC style microservices. So what we're going to do actually mainly is we're going to No, we're going to replace, we're going to get rid of the express.js entry point, and we're going to define a restate entry point instead. It looks not all that different. You know, we're defining a service here, we're giving it a name, and then the handler run, this implicitly gives us a root agent slash run. It's a bit more sort of by convention and restate. And then we're starting, you know. we're starting an endpoint where we can build multiple of these services which are basically groups of handlers and we're listening we're listening on a different port here and um you can see this gives us actually access to the reset context in addition to the request so let's actually forward this here and now basically we're serving we're serving the same thing through um through our reset entry point but that gives us a few interesting uh interesting um capabilities now so we can actually now start doing um can actually now start using durable execution facilities. But maybe even before we do that, let's just start this and bring this up. So we're studying this simple agent again. In our case now, we actually need to start the reset server. The reset server is just a single binary that we can fire up here. And then we're going to we're going to register that service at Restate, because the main philosopher of Restate is that you're not talking to the service directly, but you're putting a Restate as a proxy or broker in front of it, and that way it sits in the middle of your request and can add the durable execution. So we've added this simple agent here. Restate itself runs under 8080 now, the agent under 9080, and we should be able to actually do the exact same request again. Yeah, and we're getting this response. Can I actually see that this went through restate? We're just looking at the invocations and this was what we just did. We just get the parameters, what's the weather in Berlin, and then the response. We didn't actually do anything else. We basically just got a durable invocation here, but one of the nice properties we kind of get immediately out of this is let's say we're making this call. and we're killing this thing on the way well actually it wasn't fast enough to come to complete it but we um to kill it but you know we we can actually we decouple sort of the um the light come of that invocation from um from the actual um from the actual you know connection and request and if it dies in the middle and we bring it back it's basically all a resilient um connection the interesting bit starts actually uh starts actually now let's actually start to use some of our durable execution um magic so the the first thing The first thing that we might be interested in doing is actually saying, you know, model calls. If we have a long running agent, model calls are expensive and non-deterministic. And if it crashes after a couple of minutes, we don't wanna repeat those because maybe they're actually gonna give us a different response. We're going to go a different route. And also, you know, it's gonna take the agent a long time to get back. So what we actually do here now is we're turning where instead of using the model as it is, we're using basically middleware to make the calls durable. This in itself is if we actually go into the definition, it's an extremely simple thing. It's basically just a wrapper that says turn this into a durable step. What these wrappers look like, we can actually see here explicitly if we look at the tools, because we're going to do the same thing here. We're actually going to turn the tool into a durable step. Let me zoom in a little bit to make it easier for your calls to see. And the only thing we actually need to do here is, it's gonna say reset.run. Let's actually give it a name, fetch, or other. And then this is the function. This is now our durable step. That's actually it. And if you wish behind the durable model calls is exactly that. It's just a wrap around this. That's really everything. Let's make sure we're running this. And if we're doing, if we're calling this now again, you know, what's the weather in, let's actually take a different city maybe. What's the weather in New York like? We can actually go to restate here and it's going to tell us now because it actually really tracks each individual step. It's going to give us a very nice breakdown. Okay, the first thing we did is we're calling OpenAPI and this was... what came out of it. We're supposed to call a tool called get weather. Then we're actually calling the weather tool, which gave us temperature 23 and cloudy. Right, great. And this is the final response then that we got from the second inference call that we returned. So this is really, if you wish for just getting some durability and visibility into some simple agents. This is basically all you need to do um it doesn't mean we're going to stop the community call here because there's a lot more but this is really just wanna um so the pause here maybe for a second um this is yeah this is basically all the set of changes um you need to do it let me try and um i'll show you one more thing and let's um if you want go into the first round of questions then um Here's another agent that answers slightly more interesting problems as an example, again, taken from the Vasella AISDK. It's supposed to answer sort of mathematical questions and then pass a calculator as a tool. It's set up in the exact same way, right? Like we're using, it's just a regular interaction with that AISDK. using durable calls throughout our model to basically secure that. We're wrapping our inference, our tool calls to actually make them recoverable. And then we're serving them through the restate entry point. We're putting a bit of like schema information around it, but otherwise that's really exactly the same thing. Let's run this. This one gives us a little bit of a more interesting interaction history. So this one runs on 1981. It's a different service. I need to actually register this at Restate first. There's a bunch of agents here that we're gonna use in other steps. And we can now interact with this. I'm actually using the Restate UI now to interact with this. This is basically, you know, for any service, you can basically just go here, Playground, and then it allows you to... you don't have to go to the command line and curl all the time. Also, it's gonna infer schema and so on for you. So I've prepared a question here, again, taken from results example, like a riddle, taxi driver has this hourly work and pays amount for gasoline and so on. It uses fairly big numbers because when numbers get big, then the LLMs like to call calculators instead of like doing the inference themselves. So yeah, for sending this request, we should be able to to see this in the invocation running and it's gonna make yeah it's gonna make a couple of like a one inference call here then a couple of calculations and then goes to another i guess another inference call um so far so good one of the um i mentioned one of the interesting bits of of durable execution is um it helps you actually when agents are running for a while and then when something goes wrong that they don't that they don't really have to redo everything that they did um that they did before so let's try and um Let's try and see what that means. We're going to maybe first use a very simple problem. Namely, let's actually just pretend we have a bit of a flaky network. So let's gonna block just access to OpenAPI here. This is just reconfiguring IP tables so I can't reach it from my local machine. if we're calling this if we're calling this here again we're going to see that this one tells us actually okay this isn't quite working i cannot connect to the api unfortunately the sdk doesn't give us a super good error message but we can actually see it you know it keeps trying it's working on this um and it's going to go into into more and more back-offs actually, I configured the back-off interval a little lower, that we get a bit more interactivity, but usually it would actually get to ever increasing back-off for retries, ultimately even suspend, what that means, we're gonna see later. But it would not give up on the execution. And as soon as we actually make access possible here again, we should see, the IP tables and the connections and have to refresh in the background where we should see that this is like, you know, at some point keeps going. So this one failed for a while, but eventually succeeded. And then the whole thing succeeded. Maybe so far that's not terribly complicated because, you know, you could just have like a couple of retries around your model inference call and so on, and that should handle the situation. So let's make it even a little more challenging. Let's say we're gonna, you know, pretend it's an environment where really our processes and everything just cut crash and go away. So I have another shell here, which tries to do exactly that. If I run something here, it's just gonna inject a crash once in a while. So yeah, let's run this process again. Let's run it, like start our agent endpoint in there and send a similar request again. You should see this is executing. And at some point in time, yeah, this is going to tell us, okay, you know, I actually can't even reach the service anymore. Something must have happened there. Exactly that happened, you know, the whole thing crashed. But you can see it actually keeps working. The next retry is in a couple of seconds always. And it also tells us, okay, the problem is I can't reach the remote endpoint where the agent code is actually running. That's what Restate Server tells us. It's managing the durable request, the durable execution for us. And it tells us the code that runs the agent itself isn't actually running anymore. Let's bring it back up. And we should see basically that this whole thing that is like in retry, just basically continued as soon as it could reach the endpoint again and let her result, right? So we didn't actually do any re-execution or previous steps. It just kept... kept trying to continue from there and um we brought the code back up and and you know everything concluded um again there's like nothing specific we're doing in here other than really wrapping um the tool call and uh wrapping the the model call there's no no database infrastructure that we've brought here where we persist uh anything there's no queues we've brought in the only thing we've done is aware is implicitly acute through restate but we didn't really do anything else except putting putting the reset broker in front of our deployment and letting it accept the request, forward it to our service here and manage the execution. The same mechanism allows us to actually do a bunch of really nice things also when it comes to human in the loop, long running agents that suspend multi-agent interactions, even streams and um and persistent resumable sessions and all of this um let's go into this in a moment let me pause here just for a second to take uh to take some questions um okay so maybe one question um what was the motivation for um having to manually put the restate.run inside of the two call functions instead of just for example like wrapping generate text was there like a design decision that motivated that Yeah, that's actually a good question. Let me go into this one here, right? Like, why aren't we actually wrapping just generate text versus why are we wrapping this one here and this one there? You can wrap actually generate text. The thing is that generate text itself can be a fairly heavy weight, long running operation, right? Like it can actually, the max steps here is actually the number of loops it does. It can do 10 times a model inference call. It can invoke, you know, any number of tools in parallel multiple times. The level on which you basically wrap is the granularity at which you would say, okay, reset can interject here, track the progress and recover, right? If you wrap just generate text as a whole, it's kind of an atomic failure unit, right? And it's gonna recover that atomic failure unit. It makes total sense if, for example, you don't have max steps here because you say, okay, I just want one LLM inference and it may tell me to use a tool or whatever and then I interact with that manually or yeah, and then I'm... i'm building more of a traditional workflow of a sequence of you know generate text and then inspect the input and i might call it generate text another time and then i might actually you know depending on the input go a different branch in my code but then it can make total sense to actually wrap generate text in the in the case where generate text actually does multiple steps i just felt it's it's actually nice to to read maybe it's a little bit more on a fine granular basis so that you you know you get more fine granular recovery but ultimately really you know it's up to you it would have would have been completely viable too then we would have to redo a little more work in case of failure but if that's you know that's the failure unit you wanted fine that's okay yeah that makes that makes sense thank you There's a question, how can Reset help managing memory long-term short-term for the LLM to build the context? That's actually a nice question. Maybe we can get to this in one of the later examples. I don't have this in the main set, but we have examples for that somewhere in our repository. And if we have a little bit of time in the end, we can get to that too. Maybe the short, the TLDR is you can actually use what we call virtual objects in Reset, which are basically basically stateful versions of services, which where you can actually, where you make the execution in the context of an ID, which can be user ID or chat session ID and so on. And then you can attach state in that context, for example, yeah, LLM context or just like chat session context and so on and have that available to forward to the LLM. So it's basically a way to make It's basically a way to make handlers remember state across individual invocations. So yeah, it's possible if we use restates virtual objects and for every few moments in the end, I'm happy to show what that looks like. Okay, let me continue then and go to one of the next steps. So this is basically just like the very basics of saying, okay, we have a simple, you know a simple um simple agent that just has llm inference and tool calls and we're we're letting it become durable and handle um handle failures in recovery i mean just to make it a bit easier actually now launch this again in the shell where we don't inject failures um so frequently so um otherwise we're going to be watching retry some more than um than the code one of the one of the next things that's actually fairly nice um in in the durable execution system is you can let the system take external signals. So this here is the, the idea here is for this example is an agent that, for example, evaluates a loan request. And if you look at the system prompt, it's actually structured in a very simple way, you know. If you're trying to get a loan to develop AI technology, just like go ahead and approve. If it's below a thousand, approve. if it's above 1000 and unclear, use the risk analysis tool. The risk analysis tool, what it really is, is it sends an email to a human reviewer, and then it awaits the human reviewer's decision. And the way we can actually model this very easily is by saying, you know, we're waiting for a promise, a persistent promise. That's how JavaScript TypeScript calls it. In Java, you would say in future or in Python. we're basically just saying, okay, we're listening to completion of a future called approval, which is scoped to our agent here. I'm gonna show you what that means in a second. So this is basically how that works and we don't have to do much more. How do we set this up? We're doing something slightly different than before. We're not just defining this as as a service here, which basically just has this one handler. What we're doing here is we're setting this up as what we call a workflow in ReStata. Workflow is just a special type of service that has a run function, which is like the main work. And then it has a bunch of additional handlers that you can define to interact with the execution. And in this case here, you can see there's like a handler called approval. And what the handler called approval does is it takes that promise approval and says, I'm gonna resolve that promise that future with whatever you give me. Let's actually see what that looks like in action. All right, so yeah, there's, when we invoke, one of those workflows and restate. What we have to give it is basically a workflow key or workflow ID. That is basically the scope and the idea under which we can interact with that workflow, right? So, you know, when we approve something, which one would we approve? That's basically identified by the key. So let's say we do loan one. There's, you know, some SEMAs asking for a lot of money to develop AGI. If we send this, okay, we don't have that. Oh, I might have actually renamed that service. Okay. Give me one second. That should be okay. I think I just renamed this a moment before the demo. Okay, so here we go. It's actually called Loan Evaluator now, not human approval. Okay, great. So let's actually have SEM-A do their thing. Okay, it's approved. If we go to the invocation, there was one of these rules where it actually completes it immediately. It doesn't actually have to go through human approval. Actually, let me apply for a grant to not develop AGI, maybe to, you know, I don't know. It doesn't have to be quite so much. I'm not that greedy. Okay. It's a different workflow this time. If we send that request, we should actually see this one. going through exactly that path, right? We see it does the first model inference call, which says, okay, we need to run risk analysis. It notifies our human reviewer. The human reviewer notification is something as, in our case, it's just basically printing something on the command line. And I think in reality would actually let this send an email. And then now it waits for the promise called approval. And one of the interesting things you can see here now, it actually just suspended because it understood, okay, There's really nothing I can do at this point. I need to wait for this input. There's really no point to keep the agent alive, to keep the process up. Imagine you're deploying this on something like Cloudflare Worker, say AWS Lambda, where you're built by the second or millisecond even. There's no point in keeping this process alive where it's waiting for the promise, right? So this is what Reset does. It basically just lets it go away and says, okay, I'm going to bring you up when the approval is there. Let's actually send the approval here. for loan two or not. It's actually approved, why not? We're sending this one and we should see here, okay, immediately this actually worked. We got the result from that promise and that now we could feed this back into the next inference call and we get an approval for our loan. So far so good. So this basically lets us build with this type of human in the loop workflows by just by using In this case, we're using just a tool and a tool that, you know, it's a long running tool that suspends in the middle of its work because it depends on external data. This is one way to build human approval. Another way I guess to build it would be like multi-turn conversations, which I guess goes back to the other question in the chat earlier. You know, how can you use it to remember context? That would be the other way, and maybe we can actually look at this towards the end. There's a bit of a special case here that you can also explicitly ask Reset to make a pause. This is an agent that we're gonna use in one of the later steps. It's a very simple automated risk assessment agent that doesn't really know how to assess any risks, so basically just flips a coin. but it has a tool pretend to think which basically just pauses for 45 seconds in order to you know make it look like it did actually any work um this this um this sleep in this case it might seem a bit you know a bit artificial and beside the point but this is something you can actually use if you um if you do want if you do want to do things periodically or if you actually have like long backups let's say you have a research agent and it's supposed to like periodically try something you can actually implement this with long sleeps there's a counterpart to these long sleeps we can actually basically schedule schedule a tool or a call to yourself for the future all of these things kind of you know come come built into the into the durable execution the interesting thing about that sleep is also it's not going to be like a sleep as if it's just threat sleep it's like it's a durable sleep it's going to make the process go away and come back after after the sleep has elapsed and if it fails in the middle it's not going to sleep again 45 milliseconds but it's going to look at the point in time when it recovers you know am i past the point where i should you know wake up again or not um yes if we actually interact with this one this is going to look somewhat um somewhat similar to um So it's similar to the other one. If we kick it off, we can, let's actually see the suspension in action. I've currently configured the system into around about 15 second timeout before it goes to suspensions. So we should be able to see this exactly as the sleep keeps executing. At some point we see the system is actually gonna flip to suspension. It keeps waiting. We can actually, while it's suspended, even take the process away. It's going to, It's going to not realize that the process is gone for the moment because it's anyways assuming that this has gone away. At some point in time, it will try to wake up and restore it. And then it's going to go into the regular recovery cycle. We can also just bring it up now and actually see that this means we're not really starting everything from scratch, but we're, yeah. after 44.99 seconds we woke up we basically restored it and immediately completed um let's actually show how we can use this also to you know to compose multiple agents together let's say we have our our classic loan evaluator with a human approval let's assume this risk evaluator was actually a meaningful uh research agent that would actually go through your documents um through your credit history and so on and um make an you know make an informed decision on whether it should approve a loan for you or not. This is what we've sort of done in this example here. It's a more elaborate version of Evaluate Loan, which has two tools available. The first one is the risk assessment. The second one is a human evaluator. And the prompt actually gives it a few rules. There's certain type of technology which it always approves. anything below 1,000 approve, anything between 1,000 and 100,000 call the automated risk assessment agent. And if that returns low, you can approve. If it returns high, you need the human evaluator and above 100,000, you always need the human evaluator. So it's a bit of a, I don't know, if an actual credit company would structure it like that, they'd probably hardwire the rules more than trust on the agent being precise and interpreting that. But just for the sake of this example, assume we're actually defining our, We're defining our logic of how we want these to be handled through this prompt. And the tools that we structured, you probably recognize this one from the previous example, the human evaluator. It's again, we're sending the email, we're waiting the promise. The interesting thing is in asking another agent to do work, in our case actually looks are quite similar. We're also using the tool thing here, just because tool is the main sort of point that we can plug in in this SDK. In other SDKs, you can probably use something like a handover or so, and this here, this we're gonna use tools as the sort of, yeah, most general purpose way to extend, or should we implement other functions. so what we what we're doing here basically is when we see the loan request and we want to ask another agent to take a look at this um this is basically is an rpc call to the other service behind which we host this one um remember everything in in reset is basically hosted as a service again this agent here we exposed this um similar as before this is our workflow service with a run function that calls this one and then some auxiliary functions that you know let us interact with these durable promises The risk assessment agent itself was also deployed as just a service here. If we go to the bottom, we're gonna see here, this is exactly how we deployed it. Pretty much like the initial example, assess risk is our service, is our function within this service of this name. We're just going to call this function and here we basically just define a bit of schema. And, you know, when agents are basically callable services, we can also call them just from this service here. So that's what we're actually doing here. We're basically creating a client at what's that agent, and then we're asking it to, you know, assess the risk for us. So the interesting thing is if we're making an RPC call here, it works a little different than like classic, let's say classic RPC calls in other applications. It's really an... it's really a back and forth queuing system if you wish so while we're making the invocation we're actually putting an event for this service um to be invoked for this agent to be invoked and we we have basically a durable future on the response of that service here which allows us also again to suspend while the other agent is working and when this one is done it's actually going to complete that that future with its response and then we can we can continue and wake up so this actually looks like a sequential rpc interaction but what it really is it's like asynchronous messaging through queues and because it's uh it's integrated on both sides with the durable execution mechanism here on the center side it's um it's part of a you know often a reset operation we're executing this through the um through the reset context And on the other side, it's a restate handler. We get this nice sort of end-to-end item potency out of this without doing anything. Yeah, let's actually see what this looks if we invoke it. Should have an example here. So yeah, let's actually run here. This is our multi-agent loan evaluator. We're calling run on this. john is going to request 50 000 bucks for for a big vacation that that might be something that you know that's false exactly within the range that it goes to our automated um automated research agent and let's take a look so let's send this this request over and um and look at the invocations here what can we see here and we can see that our our risk evaluator here get exactly that result from the first invocation. Okay, I need to call the risk, hand this over to the risk assessment agent first. So what we're gonna see here is we're going to make a call to risk evaluator assess risk, you know, for our parameters. And this is actually currently suspended our agent again. So this invocation has actually gone away. There's no resource waiting because, you know, the risk assessment agent is taking its free time and while it's working, there's really no point for us to stay here. Can actually go into that invocation as well. And this is the one that was just suspending. It should actually wake up and be complete now. Okay, so if we go back to our original invocation to our multi-agent loan evaluator, we should see that this one has actually come back. it came back with high risk okay so what it did now is you know fall back to the to the next thing human evaluator and it's waiting on the promise so um yeah so this is basically a way that that we can can let a promise sorry a promise a process and overworked another process so this agent while it's working can actually say i'm going to hand hand work over to another one while that one's doing work i'm actually going to go away release my resources and it's automatically going to come back when the other one is done and um because you know these rpcs and the whole messaging mechanism uh goes through reset it's also able to basically tell us exactly like it's called sort of an end-to-end timeline of what happened so we can we can um we can actually expand this here and say okay while um while this was executing what was actually happening in the intermediate steps right like this is the sort of condensed view from just the lone evaluator and we can get the details into what was happening in the intermediate step. So it's kind of a workflows end to end, even though we're sort of actually calling across different services. And yeah, can do the same thing here that we did in the previous example. Maybe this time, maybe this time, we actually deny it and see where this leads us. Yeah, okay, not the workflow ended, denied. Okay, let me stop here for just a second and see if there's any questions about this. No questions. Either I did an amazing job explaining this or everybody zoned out. I'm gonna assume it's the first one. All right. Where is the code running was just something. somebody asking the code so in this example um i'm actually just running this as a local process this is what um this is what i started here it's just a you know it's a it's a local it's a local process because in in restate every you know everything is basically just um it's just durable functions that are served they can be served as serverless functions or they can be served as long-running processes in this case here i'm basically just defining a reset endpoint and starting this locally. It's like, think of it as if I started in a Springboard or ExpressJS or whatever application locally. I could take the exact same code, put it in a container and deploy it somewhere in Fargate, or I could use a different endpoint and then actually serve this on something like AWS Lambda or so, but generally, yeah, because it's all just It's all just like handlers that you serve behind an endpoint. It works exactly the same way if you run it on your local machine versus on some other service. I'm running it locally here so I can just kill it and restart it and redeploy it for the sake of the demo. So ReState doesn't manage containers. You deploy and leverage as a durable execution engine. That's exactly right. It's basically meant to be in a durable execution engine that works with code that runs where you usually run your code. um trying to be as unopinionated as possible and ultimately everything underneath the hood is http requests with a bit of special support for for some um for some targets like aws lambda so um that don't accept http requests directly you know unless you put api gateways in front of them um yeah any plans of having any agent-specific sdk in parallel to the restate sdk So, I mean, this is a fair question. At the moment, we're trying to basically go the route of saying, okay, we're trying to keep restate orthogonal to popular agent SDKs, because I feel there's lots of SDKs around there. Many of them were developed by folks that are closer to, let's say, the models or... They're close to how folks structure the applications that use these SDKs and they're doing a good job in developing the shape of these SDKs, the way you abstract agents and tools and handovers and guardrails and so on. And if we develop the 150th SDK, there's probably not quite as much value to that compared to us building basically middleware that allows you to run code written with other sdks on restate so that's what we're that's what we're trying to do um uh primarily so restate specific the stuff that we've written is basically it's basically this here if you look at um it's basically the following we're importing um we're importing where is that okay um actually Yeah, so we have a few packages like, you know, for example here, Vercel AI middleware. So this is a very small package that basically has a few utility helpers like the storable calls where it has a few more ones in order to, for example, deal with offloading model inference to remote processes, doing a bit of like more sophisticated error handling and so on. But it's generally like very, very lightweight middleware that you then just use you know with these sdks and it allows you to run run the code with existing sdks on restate um in theory you can of course work work with reset without any of these other sdks there's a few examples we do actually have in um in some of our repositories like the ai examples repository here has, if you wish, under advanced. So Restate Native Agent, actually a couple of agents that don't really use an agent SDK. They basically just use the tools for responses API, but they don't actually assume any form of structured agent SDK that defines loops and tools and this and that. for some reason for some cases um this this actually works perfectly well too so yeah that's a long way of saying at the moment we're not planning to build um to build dedicated reset specific agent sdk's but rather build the middle where to integrate um to let other agent sdk's perform their work on restate all right um i can give you a quick um a quick look at at one more um add one more piece which is how can you actually use restate to build sort of this end-to-end durable durable and resumable chat sessions so one of the and that goes a little bit into the answer also of context context management so let me quickly share the screen again and bring and bring up another bring up another example here okay so One of the things we really like, or I really like personally about Restate is the fact that you can very easily use it to actually build a bunch of really cool tools that allow you to basically build custom queuing, custom concurrency control, or things like persistent. persistent sessions what does that mean exactly let me let me show you this in a in a moment so here this is a um this is the next js app that is actually hosting a bunch of services somewhat similar to um to the services i've defined here specifically similar to this like initial um tools service that you know where we post a question and it was iterating and calling you know computational library in order to to answer that question and um Well, we're going to use this now, you know, one of the things is if you, this is the observability that we have here is nice, let's say, from the research perspective, if we want to understand, okay, what did actually happen while, you know, we're going through this tools, we can look at this invocation. But, you know, from the perspective of somebody who interacts with this thing, maybe they want to see the reasoning progress that, you know, the way if you interact with these modern reasoning models and they show you their intermediate steps. Yeah, something like this, so you're not waiting for minutes and minutes and minutes, you don't even have an idea what's going on. So this is exactly what we're doing in this next example. So what I've just started here is a mini web application with a chat interface and a couple of, a variant of the sort of tool agent that publishes its intermediate progress. Let's take a look at what this looks like. I need to deploy this slightly differently now because it is a Next.js app. So it follows a bit of a, a specific structure and a specific protocol. There is, oh yeah, there's lots and lots of, I actually hope this is going to work and I'm not going to, you know what, let me be on the safe side and actually start a clean server here in order to demo this. Because I think this is going to, this has some name conditions and otherwise there's, I don't know, I didn't actually implement this in a super elaborate way. And before I actually get into some of the services I deploy here, shadow some of the others, and then I'm lost in figuring out which one is actually the one I deployed. Let me just start from a clean slate here. Okay, so this is a bunch of services defined in this next JS app. If we go to... We go to this side now. We can do something like, we can do something like this. We can actually pose it, these type of questions like this, you know, this math, this math riddle here. And while it's running, it's actually going to tell us, it's actually going to show us the initial intermediate steps to which it goes, okay, what do I need to find out? What tools do I need to call? What are the results of the tool call? And what is the remainder of this? And so the interesting thing is this is, so this is basically a persistent stream back from our agent running to the browser. It's not actually just a persistent stream. It's actually even, it's a shared session, right? Like if I put the same URL here, I'm going to get the same session here, right? So it's like almost, yeah, it's like a, it's a persistent session. I can ask this question again, and we're going to see that, you know, it basically synchronously runs in both of these windows. So we have basically a pop-up here where the agent publishes messages and our interface actually subscribes to these messages. This is actually an extremely simple thing to build and reset as well. These few lines here, this is basically the gen, like 108 lines, and that includes schema definitions and everything and imports, right, is basically a persistent Pub-Sub that the agent can publish to the different clients can subscribe and resubscribe. And even if we start, if we copy this to another window, we're gonna see the complete previous history here. and they're gonna now receive updates as well. So this is sort of like persistent, subscribable, resubscribable channel is with everything with skew and import. It's like 108 lines in restate. And then all we actually need to do in our main agent is something like this. We need to actually tell it to, we need to call publish message to that. to that service that implements this persistent channel so we're publishing the initial prompt and then we have a hook here when the step is finished we're going to publish a message um you know what was the the text um what was the tools that were supposed to be called and what were the results of these tool calls and um and that's really it the pub set itself here is is a restate virtual objects it's one of these like services that maintain state and it's um it's basically an a virtually sort of infinitely scalable pub sub in the sense of like every session has its own ID, basically gets its own queue object to which it can publish and subscribe. And you can pretty much have as many of those as you want. They are completely isolated against each other. And yeah, that's it. Let me pause here again and take a few questions if you want. all right i think this is uh yeah there's a few questions here in the chat let me actually go through this there's from martin it would be cool if virtual object state was automatically pops up i think this is a really nice uh this is a really nice thought and it's it's something that we've been playing with for a very long time basically have sort of like change data capture which functionality for um for virtual object state yeah I think this makes total sense. I would even make the, I would even make this implementation simpler, right? And I think it would basically just like, you know, append this to that, you know, to that channel state and that's it. And at the moment, there's a few, you know, a few lines that actually access the state, register a persistent, like an awakeable, a durable callback in order to notify, to signal, okay, here's somebody that wants to be notified if there's anything else. because you don't need to build retries and failure handling and concurrency and so on. This is generally not too much code, but I get it. We can probably get this down to 29th if the state is sort of like automatically subscribable too. There's another question. Does restate follow actor model principles internally? That's actually a very good question because the answer is yes. Virtual objects, you can think of them actually, they're stateful actors mixed with durable execution. What we basically get... Okay, I closed the... No, we're still here. So yeah, basically each of those ones here now defines an individual actor, an actor that manages that section that you can push data to and subscribe from. The actor itself is... So if we go to the pubs up here, we're going to see that this has, for example, when you publish here, you can actually get certain fields from that actor. Here we're just getting a field called messages. We're adding to the existing array our latest message and then updating this field again. We're also managing subscriptions here. And each Each of these methods is basically a method, is a function on the actor that follows actor semantics. So they get an isolated view of the actor and the future functions are enqueued. So they execute after the ongoing one is done. But each of those functions itself is like a durably executed function. So it gets workflow semantics during its execution. I think this is where it becomes really powerful because now you're actually getting the statefulness, the lightweight concurrency of the actor model with the resilience and composability of the durable execution model where you don't have to worry about anything failing in the middle and does that leave your actor in a partially incorrect state. you know communication with that actor is guaranteed to be end-to-end reliable and idempotent exactly once um all those niceties that you get out of a durable execution system um so basically integrate into the actor model it looks like most of the other questions were already answered by my colleagues um in in this in the chat there's um There's a few more questions around like, how do you version long running workflows when it evolves? This is actually a fun topic in itself, but actually pointed to the documentation. I think if we go into the details of this, we're gonna blow this meeting, but I can tell you as much as that there's dedicated support for that in Restate with keeping different versions of keeping different versions of workflows and durable functions and yeah, and understanding on which version of workflow started. So you can actually pin an ongoing execution to an ongoing code version. So you're not getting the problems of like non-deterministic changes that break durable execution while moving new invocations to newer workflow versions. I'll paste it a link to the documentation in the chat that actually goes into a few more details. Thank you, Martin, for the kind words. they said it's very cool probably my favorite implementation pattern for uh i've seen so far for agents thank you it's mine too all right um if you folks want to stick for just a just a moment um then i can i can quickly just show one one last bit um about you know like state management for sessions as i mentioned earlier um and we can we can use the same the same examples here so um This is, let's say this is a chat agent where we're using restate to manage context and memory. So it's an extremely simple one, right? Like the only state it remembers is basically an array of core message. So everything you've asked it before. And it has one function, which is message. And what it does is, it actually other than you know like doing the same setup as before we're taking the um we're wrapping the model to make it durable what we're doing is we're actually going to get the messages from the from the state we're taking the messages and we're concatenating the next user message to it and then we're feeding this through our you know text generation and then before returning this we actually store the response instead again so we're taking the current messages plus the new messages that come out of the response that is it that That way we can actually let the system remember things. So let's actually look quickly at what that feels like when we interact with it. So if you're interacting with a virtual object here, which is like the mechanism to have these sort of like stateful entities. It looks the following way. So we're interacting with restate chat. Then you can see the key here, message. The key is basically the name of the object, of the actor, of the session that you're interacting with. So for example, if I'm putting my own name in here and hi, I'm Stefan, nice to meet you. And okay, what? And then I ask it, what is my name? I should actually know that because we're letting it remember it. I don't seem to be getting an answer now. Okay, very good. Let's actually use this to debug this. Use the reset UI to debug this. Okay, let's actually see. So this is chat. This one is still running. Why is this one still running? We're calling... We're calling the model and the model is not responding, but it's also not throwing an error at this point. It just seems stuck. Let's actually see if we can get something out of this one. Okay, I might have actually killed, messed up the route while I was preparing for this demo. Okay. So sorry, that happens during live coding. I removed a lot of these rules here, unfortunately. Might have to rebuild the application in order to make it work. It's in a bit of a weird state because I think it's relying partially on a compiled cache and partially not so, apologies for that. Generally, if you interact with the system, if you interact with virtual objects, you always have to basically provide the name of the object that you interact with, and then this actually remembers the history. If you give me a few seconds, I can rebuild this, and we should be able to get this running in a moment. Let's actually take a look at a few questions. Okay, what are the next few themes you're looking at for additional AI LLM agent support? How is that question meant? Like, what are the next patterns we're trying to build, to add support for? Yeah, okay. So there's a lot of directions. There's a lot of directions to go. I think the first one is that we do definitely want to wrap up a lot more of the utilities that we have, for example, in these... Yeah, and these examples here, like the PubSub, the way to manage a shared session and so on, just like put this into this like middleware library. So you can basically just say like, okay, let me import a shared session and then I get that, or it pops up and then you basically get this sort of cues. Because I feel a lot of the work when you build these applications in a bit more serious way, it goes actually into the plumbing. Like how do you actually manage the interaction between the agent and the human? Let's say the... the more sophisticated version of this loan workflow maybe we can actually do a follow-up meetup or we can go through this where you would actually have um we actually put put um um it's more of a claim processing agent actually where you put a claim there and then you have an energetic subset that says okay i'm going to research all the information around this claim and i'm going to ask the user to provide this and that if it feels like this is an incomplete picture and so on, like all this thing, all this like back and forth between the agent and the user, right? Like it's not, it looks super simple in most of the examples. Let's make the multi-turn conversation or whatever, just send it and like wait for a message here. But like if you actually plumb this together, it's where you can, you can easily spend 80% of your time on this, but this is actually one of the strengths of a system like Reset, right? It makes actually this sort of like connectivity pieces, these plumbing pieces are very nice. This is definitely one of the areas where I think we should publish a few more utilities and so on. The other bit is going more in the direction of... These examples were simple in the sense of they basically just kicked off one agentic loop and then they had a suspendable tool or a tool that caught into another agent. But a lot of the, I'd say more real world examples, they're not... okay, here's an agentic loop with a super big prompt and a set of tools and now go do your work, right? Like I think the really successful, sophisticated agents, they're like to a good extent imperative flow where you, you know, maybe make a few control for decisions based on classifying input through an LLM and so on. Then you may have like a micro agent that is like a step in that workflow. And these things actually work really nice with the durable execution system as well, because it's not like heavily tailored to just agents. It's actually something that generalizes beyond them. So it's really good at sort of mixing the more traditional workflow logic like with. with agentic logic right so you don't you don't have a workflow for your traditional logic and then an agent and then like this is causal to this there's actually just like one single workflow in the end it's just some of the steps are imperative defined some of these steps sort of like the agent comes up with them as it loops and and that actually i think is the real power of the office system like this it's it's a bit more it's not the hello world agent use cases i think when you try to when you start to to make it real, start to integrate them with your existing workflows when you, yeah, when you build the sort of like connectivity with the other systems with, you know, back and forth with the user. I think this is where it becomes even more powerful. And I think that's a direction I would like to strengthen in the future. But, you know, definitely happy to, I mean, these community meetings are partially for us to hear what would you like actually like to see? Like, where do you feel like this could really use better support? Yeah, that actually resonates because one of the things I was looking at earlier was like trying to, you know, wrap my head around sort of like the mental model, you know, for restate. And like one of the things that I was trying to process was, yeah, like is an agent a workflow or a virtual object or a service, you know, or some like in restate terms or some combination sort of of all of those things. and maybe plus one to another question or comment earlier in terms of like how that all relates to like the context management um and perhaps even being able to have more more visibility of the the constructed uh context um in in the restate ui as well you you can actually have that let me let me try and get this uh this other example up and running and i'm sorry that this is a bit of a that was not one of the prepared parts for the demos um to reconstruct a bit of a in the middle sort of dev ongoing code snapshot. Let me see if it works this time. Just recompiled everything. So yeah, this is actually going to work now. So if we go to the chat session again here and Yeah, so let's say my chat session is session one. Okay, we're gonna get this response here back. So you can, yeah, so we have this, all sorts of invocations. Okay, we're still pulling from this channel and we're not finding this. Okay, this is why all these failed invocations are there. Ignore them for the time being. If you go to state, you can actually, we can actually see exactly that. If we go, for example, to chat and query this, we can actually now see session one. If we go there, this is basically all of our state. Yeah, or here we can actually see it even nicer. This is basically everything we stored in there. And if we do session two, somebody says, I am, I don't know, Jane, then... We're gonna see, okay, we actually have these two sessions going on now, right? Let me now ask Jane, hey, do you know who I am? In session two, I don't have access to a person individually. Since you mentioned your name is Jane. Okay, that's all I know of you so far. Fair enough, that's a good answer. So yeah, in this session, we have done exactly that context. So yeah, it actually knows that. this is it, and if we ask somebody in a completely different session, do you know who I am? We shouldn't actually have anything, no. Don't know anything, and we should see session three has actually popped up here now. So you can get a bit of that, you can get some of these insights. I guess what is actually something that could be fun for one of the next things is trying to Yeah, trying to group these things a bit together. If you're building sort of an agent template, then to say you have the invocations connect from a specific agents together with that agent state sort of in a view side by side. You do have all this data right now. It's just on different pages. There's invocations versus state. I want to say though that like the experience should be pretty customizable because Reset's backend is just SQL actually. So you can always, You can always sort of build your own view by just issuing these queries, right? We can actually see there's a bunch of chat state for session three, session one messages. There's the values in binary and UTF-ed. There's also the PubSub, which currently holds only a subscription because we have the other page open that we're still pulling, even though nobody was publishing there. And it goes actually beyond that, right? Like we can add this. We can also do... You can also check this invocation for the execution progress, right? And we can see, okay, from all the virtual objects, because the only things we had were actually, yeah, PubSub and chat, which are all objects. Those were the invocations, those were the journals and so on. So you can kind of access that, and if you want customize, I guess, the UI with a little bit of, can probably ask, can probably vibe code a different UI that actually puts this side by side. Maybe that would be actually a good project for us to do at some point in time. Just use that, you know. Yeah, that would be very cool, because that would kind of, like, albeit the need for, one, like, the general, like, open telemetry style tracing and spanning, but also, like, the LLM-specific observability tools as well. OK, perfect. We're actually way over time. I'm not sure if you folks want to hang around a bit more. But yeah, I'd say, from my side, having to wrap this up for today, I feel like we should do another one really soon, because there's so much follow-up. like uh we can go a little more into how do you actually build an agent that uses sort of a separate session to interview a user and collect more data um how do you you know different patterns for like human um human in the loop use a virtual object to maintain state in a multi-turn conversation versus you know make this a durable future um no pros and cons for for both approaches there's there's so much there's so much we can do the interesting thing is we've we've tried to build the foundation of restate as this like very orthogonal and general building blocks durable execution you know virtual objects as faithful actors um you know workflows durable futures and promises um you can kind of compose things in the way that feel most natural to you you don't have to follow an exact pattern if you want to to say okay i'd rather have the agent feel like a long-running workflow and send a signal in there as a human approval then you know this is the pattern we used here if you feel like you want to do a multi-term conversation just make it a stateful object remember the chat history and push new messages and concatenate them there you don't need to do much um much around it um yeah it's kind of up to you um yeah i feel as this agent space is evolving so much that we want to probably preserve this for a while and maybe at some point in time when um when certain patterns sort of evolve as, okay, this is the textbook where you're supposed to do this, then maybe we're gonna go and create a few more opinionated patterns for this. But at the moment, I feel it's kind of helpful to say, okay, wherever these SDKs take you or wherever the design patterns take you, should be able to just map this to the runtime rather than being forced into a specific structure. Because that's one of the biggest... I think criticisms I've heard of most of these AI SDKs is if you go to something really sophisticated, often people kind of sort of walk away from these SDKs because they start to narrow you in and how you design your software and your application. And at this point, we don't feel we want to do the same thing with ReState. Maybe that's a good final sentiment for this community call. All right, thanks everyone for joining and for staying over time. See you around for the next one. Thank you.

​In this community meeting, we show how combining Restate with your agent SDK (Vercel AI, OpenAI, ...) gives your agents a set of powerful features:
🔁 Automatic Retries & Recovery
🧠 Resilient Human-in-the-Loop
🕹️ Agent Orchestration
⏳ Long-Running Workflows
👀 Built-in Observability

​Stephan Ewen (founder) gives us a demo of what this looks like, how it works, and how you can get started!

Website: https://restate.dev/
Blog post: https://restate.dev/blog/durable-ai-loops-fault-tolerance-across-frameworks-and-without-handcuffs/
GitHub repo: https://github.com/restatedev/ai-examples