Restate 1.6: more control, more clouds, less tuning - Community Meeting February 2026

welcome everyone thanks for for um coming we didn't have a community meet up for quite some time i believe um but today we have good news to share which is that reset 1.6 is out and i want to walk you through some of the highlights um not everything because that's for that we we don't have the time but the highlights and um yeah of this release So the 1.6 release builds on the work that we started in the 1.5 release and gives you even more control over your invocations. Makes it simpler to run reset across different deployments and reduces operational overhead. And that's something that I will show you in the following slides. Just as an overview, so I will start by giving you this overview. cool new features of reset 1.6 then my colleagues nick and pavel will show you some of these features in action i think that's going to be the exciting part and then at the end of the day we have a bit of time for questions to answer them okay so the high level overview overview what's coming in 1.6 is on the slide so the the these four themes we will go through each of them in detail um but it's not all of course that that is included in the release so there are plenty of smaller improvement and fixes that that we won't mention so in general like as a one thing that you that you can take home is that this releases touches everything from day-to-day developer experience to making your your production operations easier so let's start with the better implication controls That is something that, as I've said, we have already started in the 1.5 release with the restart as new feature, for example. So usually if everything goes all right, you don't have to interfere. But sometimes your invocations fail for some reasons. And in some of them, it's actually desirable that you can have control over them. can can stop what's happening to investigate and and like repair by default when when something your location fails um reset will retry um and but for example if the service deployment is accidentally or on purpose decommissioned or some some third party service isn't available then retrying wouldn't wouldn't really solve the problem because it's not like a transient, ephemeral issue that can be solved by it. So the only way to handle the situation previously was that you had to kill your invocation and thereby lose all the progress that the invocation has made. This now is no longer needed. You can now actually pause individual invocations, then investigate what's the cause of the failure and potentially repair it for example by redeploying your service deployment and then resume the location without losing any progress you can even change the the deployment on which the location should continue in case you you register the deployment under a new name and this you can do by the cli the rest api and the ui there is one important change in the context of of retrying which is that we change the default behavior of the retry policy from killing your invocation when it has depleted the maximum number of attempts to pausing it. This will allow you actually to now go and take a look at what's wrong and then decide what to do, whether to kill the invocation or whether to repair it. This is actually an important behavioral change that you should be aware of. And this also manifests into the way you configured it. Before, you configured the retry policy under worker.invoker.retry policy, and now it's under invocation.defaultretry policy. And again, the big change here is that when you exhaust your maximum number of attempts, the invocation will automatically be paused, not killed. alongside that we also changed the the default number of max attempts and the max interval if you need to well because you you started to got accustomed to the old behavior you can of course get the old behavior back by setting the um the like on max attempts behavior to killing or to kill but this you have to do explicitly now so in case you didn't configure any retry policy yet um you would have to do that um you can do this like configuring the retry policies on a per surface surface basis on a handler basis or even from sdk that's fine okay one of the cool new features is restart from journal prefix so as i've mentioned initially reset 1.5 introduced the reset as new feature this was useful to rerun an invocation that they completed successfully or not before, but you wanted to rerun for some reason. But this always meant that you would redo all the work that has happened before. And especially if an invocation failed, this would have meant to potentially redo some extensive work. So take, for example, this agent with a human in the loop example, where um let's assume we we our agent is able to run a couple of iterations and then as part of its tool choice he reaches out to a human to approve something and because the human was already out of office it took a while like to the next day and so potentially an expensive operation and now after the approval has has happened and the agent has continued something goes wrong you you you failed because you like the agent failed because it called like a buggy tool um so in that case ideally you you would like to to restart after like by by keeping all the successfully completed steps from before this you call this fail tool and um with restart from prefix this is now possible you can basically specify the journal entry index from which you want to restart and everything from before will be preserved and that yeah that i think is a very very useful feature especially when when your workflow your service has like expensive individual steps note that um to use this feature you need to use the the latest sdks and it's it's currently and you can can use it from the ui or using the rest api the cli um the cli it's not not yet supported um but it's in any way nicer to use it from the ui as you will see in nix demo all right one of the um yeah um annoyances that we heard from many users was um like if they managed to to move um many of the invocations into like a state where where they needed to be killed canceled or or purged for example and if they accumulated many of those invocations it was painful to batch modify these invocations like before with the cli you could already do like cancel all vocation of the service but it was like a very inefficient and b um prone to failure because it was like it would fetch all the invocations and try to to cancel all of them at once which at a certain scale would fail on the CLI side. This has now been improved by properly batching those requests and sending them over to the server so that these operations like batch modifying invocations become a lot more efficient and then feasible. And this is now actually also possible from the UI. So you can select a couple of invocations that you would like to cancel or resume and just triggered from the UI and we'll do it for many, many thousands of locations now without you having to manually script around our REST endpoint. Then, one of the big improvements on the observability side of things is that we now persist error events. and show them in the timeline. So before, these error events that triggered retries in the invocations were only kept in memory and also not properly displayed in the timeline. So there was no way to inspect what has happened after a partition failed over to a different node, for example, or after invocation completed. Now, since we store these error events, what basically triggered retries um in the journal and you can can inspect them um easily in the in the um updated timeline view and um yeah they are also part now of the sys journal table so you can also use like the sql introspection api to to acquire for these error events um if you like um so this is i think very very nice to also like um and i sent after an invocation completed um was there something that i can improve was there like uh maybe um an unreliable like unreliability in my system somewhere um yeah just um like more information about invocation execution is is always better and then um one of the the um strong um value propositions of reset is that because of the all batteries included philosophy you can run it anywhere also on any cloud but there was a little bit of fine print there which was that um for for taking snapshots that you want to be able to trim your log and to limit the replay time in case recovery um happens you you have to configure a snapshot repository. And this snapshot repository needed to be an S3, or at least of an S3 compatible API. And so it was a bit, let's say, not super straightforward to deploy Reset on GCP or Azure with a snapshot repository. You always needed some form of an S3 compatible proxy. Now, this belongs to the past. because we added a native support for the Azure Blog Store and the Google Cloud Storage. So when configuring your snapshot repository, you just have to put the AZ and the GS URL schemes in front of your path and of course add your credentials that you have access, but then you reset N. create snapshots and store snapshots to those object stores which makes it a lot nicer to employ reset in these these clouds all right another like a category of of making operations a bit simpler is making memory management easier or automating it so in order to get most out of restate you have to carefully tune your memory. And this has always been a bit tricky because you have to take into consideration how many partitions there are on a node. And actually, before 1.6, there was only this option, like you told, we said there are so and so many partitions and we said, like distribute the available memory across these partitions. On a single node, that's fine. But on a multi-node deployment where the number of partitions can can change for example if if a node crashes then some of the partitions will move to the alike nodes um so that they they run more partitions you always have to to plan conservatively here so you always had to assume that the maximum number of of partitions run on a single node in case all the other nodes crashed for example and so that for the normal case you would always leave some of the memory unused now this is also belongs to the past because reset will automatically reconfigure the the memory depending on how many partitions are running on a node and yeah that way you will make most of of your available memory at any given point in time as Reset will adopt and adjust. We added also a few improvements which made it actually possible to reduce the default memory configuration of Reset quite a bit while maintaining the same level of performance. So instead of like giving six gigabyte of memory to RuxDB, by default it's now two. um and um yeah for the query engine we we reduced it to one gigabytes from four um one quality of life improvement is that when running on linux or when running yeah in linux containers reset can detect if your secret c group limit and the works to be memory is misaligned so that it can warn you that um you you might or should adjust your Yeah, port configuration, for example. Or the reset configuration. Okay. There was one, let's say, operational difficulty. Another operational difficulty was that we only had a single way to configure snapshots. You could configure that after so many records, I want to create a snapshot. But in some situations, for example, I think if there's a period of inactivity, this would mean that you wouldn't create any snapshots, or if the activity decreased quite a bit, your snapshot would be spaced out a lot farther. So instead, people were asking whether we could have time-based snapshots, which people usually prefer, because that way they can kind of bound the time they have to replay in case of a crash and now you can configure like the snapshot interval under the worker snapshots options which tells you create me n every n time units snapshot if you configure the snapshot interval too small then with reset 1.5 and before you could easily end up with with many many snapshots on your s3 and now like azure blog store and google cloud storage um using using upspace where whereas only the latest snapshot was relevant for for research to recover so what people usually did is they they manually added some cleanup scripts or used some life cycle policies that the snapshots would be cleaned up eventually but this was always a bit let's say, tricky because these lifecycle policies didn't know whether those snapshots were still needed or not. With 1.6, there's like an experimental feature that it can enable by setting the experimental numretained field under snapshots, which will instruct restate to manage these snapshots. So to only keep the latest... and snapshots and remove all others thereby you will no longer accumulate a lot of snapshot state on on s3 or so making operations again simpler then we've reworked the deployment api to make ci cd pipelines and general deployment operations more robust and less shooting yourself into your foot so the biggest change is that the fourth flag is like when doing like deployment changes is no longer by default true but now it's false so you can't accidentally override the production deployment re-registering the same deployment is now idempotent meaning it returns the 200 status code and if so if you would like have a retry in your in your pipeline it would no longer fail there is now like a new breaking flag that it can pass to the deployment rest api to tell the system that you explicitly want to apply some breaking changes for example renaming a service handler or changing the i think the service type and that is now like you should use the breaking flag flag if you want to do these kind of operations now and it is now possible to update values of the deployment like tokens or endpoints without a full re-registration there's now the patch endpoint the patchwork which allows you to change deployment configuration if needed and a nice little thing is if you use the github actions workflows that we offer them like it would automatically capture in the metadata of the deployments like information about the repository the commit and the workflow info so that you can can link a deployment back to a specific commit in your repository all right those were the the say the highlights now a want to spend like two more minutes on on some of the making you aware of some of the breaking changes so that if you upgrade you you know that that some things i have to watch out for the first thing is that we deprecated um that that the deprecated sdk versions um like like sdks that are like below the specified version here like evocation for these sdks are now rejected so you would see an rt20 one-time error when sending like an application to an endpoint that is still running java java sdk is more than version 2 for example this doesn't affect like already in flag locations but you will no longer the cluster will no longer accept those so um what this this should tell you is if possible please update your sdks because this is a preparational step for deprecating some of these older or dropping support for these older sdks then as initially mentioned the retry policy config option changed now like we dropped the worker dot invoker dot retry policy and it's now fully replaced by the invocation dot default retry policy also the default behavior as i've mentioned before Then the HelmShard resource defaults have been changed now from three gigabytes to eight and from one CPU to six to better align with the default values of reset itself. Because before, if you use the defaults, it was so small that the reset process, if given enough work, would have boomed the pod. If you want to keep the old values, you have to now explicitly set them in the values file. And last but not least, we corrected some of our Prometheus metric names. There are some incorrectly named metrics that have an incorrect count suffix, for example. So please check the release notes out for details in case you were using those metrics. update the dashboards and alerts accordingly and with that we are one slide away from demo time so hang on so yeah you get all these features by using the latest version you can upgrade seamlessly without the need for manual migration just please remember directly go from your 1.5 reset version to 161 skip 160 if you can and yeah for more information about what else has been changed in the release follow or look at the release note that's something that we started doing as of this version or read the blog post and the first way of course to get started is using the reset cloud where you already get the latest version if you start a comment there and with that i would like to hand over to nick i think for the first demo and thanks still and let me share my screen hello everyone thanks for joining us today and i hope everybody can see my screen Perfect. So we are excited to show you some of the features still we're talking about in the UI, about the invocation control. So I'm running the real estate 1.6 in our real estate cloud environment. And let's dive in. Let's go to the invocation tabs. And I can see the bunch of invocations that are paused. as he mentioned the default behavior is after maximum attempt they will be paused so let's go dive into one of them and here i can see let me zoom in a little bit so you guys can see so in this instance and we are getting a token expire so the pause happened after token expired error It seems here we were doing a downstream call to an API that required a token, but that was the token expires. Here we have a compact view and you can also go to the detail view. And this is the part that he was mentioning. So you can also see the transient error that was happening. So apparently for a period of time, we were getting the service unavailable error. Due to that error, the token that we got from the previous error now is expired. And now we are in this situation that we cannot make any progress. So our API, our invocation is always going to fail. So I can resume this like here. I can resume this and it's going to run, but obviously it's going to still fail. It's not going to help. So I'm going to pause this and it's already paused. so one thing that we can do here is we can kill the invocation so we know that the invocation is not going to make any progress and one of the features here is is one of the paper costs that we used to have is if you are this dialogue confirmation dialogue is a pain it's bothering you so now you can save for the next time you don't need to ask me to confirm and just do the confirmation so now the confirm now the invocation is killed and now we can restart the invocations and as part of the invocation restart we can actually say from which step to restart so for example here um we and the first step we were doing some heavy let's say heavy computation step and i don't want to basically uh in a new invocation to repeat that steps so i want to start to get the authentication token and just make my api come and for the purpose of all demo i'm gonna register a new deployment that basically my service is gonna pass so bear with me so i'm gonna use the cloud tunnel here and i'm gonna register a new version of this deployment localhost 1982 and hopefully this should go perfect so i have a new version of this so if i go back here to my invocation here and if i do restart that new also not only i can say okay start after the heavy computation step i can also say okay use the latest deployment okay so you can also say if if there is an issue and you fixed it then you can also say which deployment used to basically restart this invocation so if i do this hopefully this should run and voila my invocation pass everything's successful so this is this is one flow of how it gives you flexibility to manage if any issue comes across basically the invocation that you are running and to and let's go back to the list of invocations here i can see there are quite a bit of failed sorry pause invocations okay and if i go to one of them for instance i can see and there is a bad request happening we are making some api call and there is bad because probably there is a bug and we like to fix that okay and i know that i just registered a new deployment and for this purpose of this demo assume i fixed that bug in that new version of my service and let's try to resume this to make sure if i choose the latest one let's say try that for this specific invocation and see if it's working perfect it's successful so we know we fix that okay but now the issue is that all these invocations that is paused okay and now i can okay go individually select one and then come here and say resume but there is also a better way you might In this instance, I have about 100 invocations, but you might have tens of thousands of invocations. So you can actually update your query here. And in the query, let's make the query, and then you can come here and say, okay, I want to resume all the invocations against this query. So my query is basically a specific service. And deployment, I'm going to use the latest deployment and I'm going to say, okay, resume. And this should basically resume all the invocations and everything's successful. So it should help you with managing, you know, dealing with a huge number of invocations. And as Till mentioned, you can cancel, you can pause, you can resume, you can kill. Just for the purpose of this, Demolus also pairs. all these invocations just make it a bit cleaner and this is how we can also do batch operations and other than that i think yeah that that covers my demo and i'm happy to answer any questions if there is one and then i will pass it to pavel maybe one nick um so resuming on the latest deployments You kind of need to know that the steps before were the same, right? Yes, yes, yes. All right, let me take over. Thank you very much for that, Nick. That looks super useful. I'm looking forward to using some goals. I wanted to show you some of the new snapshot features that we've added in 1.6. So as Tom mentioned, we've added support, native support for the Azure Blob and Google Cloud Storage. services that we can now run natively. I have here the command line to restart 161 server. While that's starting, I just want to show you what the configuration file looks like. So this is a server that I've already, as part of my environment, exported because it's running on my machine, exported a path to the service principal key. that I have on my machine. However, if we are running this restate server in a container, for example in GKE, they will automatically infer the pod's principle based on the way we are running. We've done that. This is a brand new server. So what I will show you is it's pointing at this bucket. It's a Google storage service bucket that's running near me. and we don't currently have any services deployed so just start a watch command here which is going to list the bucket and i've already in another terminal started a load test service that we often use these so it's deploy this and i'm going to send some loads at this reach state server so this is going to start generating some traffic and what we will see is that as we start getting traffic we're going to start seeing some snapshots get created by the different partitions. So in just a little bit, here we have a snapshot for partition zero. There are only four partitions in this server configuration. I just wanted to make it very simple and easy to follow. But while that's going, let's keep the snapshots adding up. I want to draw your attention to a couple of things in the configuration. We have set a very small number of records interval just like historically we've had. The trouble with configuring snapshots to snapshot to trigger based on number of records is that if you have very large positions in your surfaces, if you send very large petals for example, very few snapshots records could lead out to very large snapshot data accumulation. So now we have the option of combining a relatively small number of records interval with your desired snapshot interval. So we could use these things independently or both together. And when they're both used together, this means create a snapshot once every 30 seconds, as long as at least 100 records have been committed. So this gives you a way of ensuring that snapshots will happen However, during periods of no activity, we're not going to be generating excessive usage on the object store. And now we should be able to see that in this bucket, after some time has passed, we have, although multiple snapshots have been generated and recorded, let's try and list this thing again. We can see that we actually have just two different snapshots and these two different snapshots are actually a moving target. And the reason why we only have two and these snapshots will be deleted all the time is that we have configured the new experimental feature for snapshot pruning and we have capped the number of retained snapshots at two. Just bear in mind, if you do activate this feature, historic snapshots created before this feature was first used are not going to be managed by Restate. So only once you activate the feature, new snapshots created from that point onwards will be tracked by Restate for the purposes of printing. Hope that makes sense. I think that is it for the demo of snapshots. But what I also wanted to very briefly mention is a couple of improvements on the documentation side. If you have a question, please just speak up. Sorry, there's a misclick. Oh, no worries. I also wanted to show you that we have made some improvements to the snapshot documentation. So traditionally, this area had been a little bit obscure. We've now got a lot better guidance on how to configure automatic snapshotting, how to think about the frequency with which you may want to take snapshots. And just bear in mind, snapshots are not backups. The state also talks about backups and it talks about retention. While I have your attention, just want to mention, we've also released a new version of the ReState CDK toolkit. If you're deploying to Lambda, this is very commonly used. This now supports, it has caught up to the ReState operator in that it supports pruning your drained deployments. So one of the really nice features about deployment to Lambda is that you can cheaply create multiple versions and you can just retain lots of old versions on Lambda. However, historically we never... proved those deployed all the versions from Restate and that could lead to accumulation of, firstly when you do Restate deployments list, you'll see a whole bunch of endpoints that you probably don't care about, but it also puts your main latest store with additional deployment information. So this is now there, we also support the new backwards compatible breaking changes deployment in Restate deployments. And then finally, I want to give you a sneak peek of another new feature that we have coming. So this is more of a Restate Cloud. I know today's about the Restate 1.6 release, but recently we've been adding some new security and enterprise-friendly features to Restate Cloud. We now have a new identity provider where we support corporate SSO. And in this case, you will see the new the new restate sign-in UI, which of course now my password manager has locked just in time for the demonstration. I'm going to sign in with this fictitious user, this is our dev environment by the way, and I'm deliberately there to show you something that is not yet in production, so we're going to say continue with SSO. If you have a corporate domain registered with ReState Cloud, this will bounce you over to your corporate identity provider. That was Okta where I've already signed in, so we've been automatically directed back to ReState Cloud. The other exciting new feature that will be offered on enterprise ReState Cloud customers is that once you're on board with a single sign-on, we now support corporate directories and role-based access control. and this will be soon rolling out to production. And what this allows us to do is create organizational administrative roles based on corporate directory attributes and these restate organization users will additionally have the ability to manage permissions. So you will then be able to list all the enumerated users that come from your corporate directory and you'll be able to assign them individual access. So this is going to give you a lot more flexibility for managing shared environments that need to be aimed at specific stages of your deployment or teams or different product configurations that make sense in your environment. That is all for the demo.

Restate 1.6 gives you even more control over your invocations, makes it simpler to run Restate across cloud providers, and reduces the operational overhead.

​In this community meeting:
- Till Rohrmann gives us an overview of Restate 1.6
- ​Nik Nasr shows us the new UI features for controlling invocations.
- Pavel Tcholakov shows what's new in the Restate runtime and Restate Cloud

Learn more: https://www.restate.dev/blog/announcing-restate-1-6