Distributed Restate Preview - Geo-replicated State Machines

Let's now look at the distributed failover behavior of our scenario. We're first going to inspect the restate cluster using the restate cdr command. What it shows us is that we're running a three-node cluster, nodes 0, 1, 2. We've named the nodes US-East1, US-East2, US-West1 based on the region they're placed in. They're listening to an address. that is in the local VPC but the VPCs are peered and the nodes talk to each other. We can see here that the log configuration for that cluster contains 24 log partitions. The sequencer here tells us which node is the leader for each of the partitions. So we can see for some of the partitions node zero is leader, for some of the partitions node one is the leader and all of the partitions place three copies on node zero one and two if you have a larger cluster the node and the copy set for each of the partitions would be different but since this is just a three node cluster each each partition is replicated to all of the three nodes below we're seeing the configuration of the partition processors which are the workers the ones that consume the events from the logs and that trigger invocations update state handle journal entries and so on and we can see there's always three partition processors for each partition out of which one is leader and one is follower and they're placed on always on node zero one and two so we're basically replicating the log but also the followers and the index state to each of the three nodes and the three regions in the setup again you can see the leaders are spread between node zero one and two so all over the cluster we have a pretty balanced setup here Now let's run a few orders against our distributed setup. Let's issue a single order first that sort of hops between regions. So we can see it goes through the sequence of creating an order and adding a few order items. And some of these calls go to west one, some of them go to east one. This script here, it has an artificial delay. The backing services add a second for each order item. So that gives us later a chance to actually trigger failures while it's executed. If we didn't have the delay, it would actually be too fast to really go for meaningful failure observations. So these scripts also verify the correctness of everything that happens on the restate side, on the server side. So they're making the requests to west and east one, but they're also fetching state back and verifying that everything actually is as they intended it to be based on the sequence of commands that they sent. Also remember, we have the verifying service in the background that implements the inventory assets tracking and the order status tracking. So this actually verifies. as well that things are consistent we would see errors if we ever see a consistency violation all right um so we can see actually one order goes through the system pretty well even though we're targeting hopping and ponds between east and west and there's no there's no unexpected sort of state um caused by something like an asymptomatic application window or so where you know heading east after heading west um would actually mean East doesn't see still outdated state. And then we're writing to East and we have sort of like last writer wins and lost update scenarios. Nothing of that happens here. Let's make this actually even more challenging. So we're now triggering two parallel orders or two parallel clients for the exact same order, always trying to do the same operations against East and West. So we're seeing this one is actually calling create on the order on East. This one's calling create on the same order on West. One will always understand the order list was open through the request and one will actually get an error that says order was already opened. Then both of them will add an order item and both of them will see both order items in the list. So in this case, we added some US dollar and some Vanguard ETF. Both of them see the same thing. Now comes one of the crucial points. One of them will always try to execute the order, where the other one will cancel the order. And we're sending those requests concurrently. So it's really a tricky race. Like, whichever one hits restate first will sort of order before the other. Even in such a situation, if one of the clients says, I want to cancel the order, ultimately, the order should be canceled. The question is just does the command to execute the order reach the order first, which was the case here. the close command reached it and the order was executed before the cancel command reached it and the order was reversed. Both of the clients ultimately actually see that the order was reversed, made it executed and canceled afterwards. Let's run this again, see if we actually see the opposite state again. We're seeing again, you know, order opened was already opened, seeing two order items, both of the clients see the exact same thing. It looked like here, okay, the command to close the order to execute it actually beat the call to cancel the order again let's execute this one more time maybe we'll actually see the race condition play out the other way around this really is just like millisecond race conditions so in this case we actually see the cancellation one won the race so you know order was actually cancelled and the execution saw that the order was already cancelled So again, even though we're concurrently heading to geographically distributed endpoints, we're always seeing things sort out in a consistent manner. There's under no circumstance any sort of inconsistency introduced by async replication lag windows. Let's now actually look at how things behave under failure. So to illustrate that a bit better, we have a few orders executing in a loop here. In the window here at the top, we can see one of those hopping between east and west regions. So making a call to east and west randomly. We have one loop executing orders against east and one executing orders against west. So let's actually wait for a nice point when maybe east is just adding a few order items, and then we're going to kill our east cluster. That worked well. Good. Let's actually see what happens here. It was happening at a very interesting point. So we shut down the nodes in the east region just at the moment when this was trying to reach an order. So we actually see a message that the connection was closed before the message completed, which means we don't really know did the order make it through or did the order make it not through. We can see that... Restate retries against West1 and the order goes through. And when we actually verify everything, everything is correct. And that's just because Restate understands how to do item potency of requests. The clients here implicitly just add a random number to each request that on retries, they keep it constant. And then even though you retry against different regions, Restate actually deduplicates based on these request identities, item potency keys really. So we can also see the other nodes in west just keep operating. We see in the top window here, whenever it tries to make a call to east, that one obviously fails because we just shut down the node there and then it retries in west and keeps hitting west and things just continue. You can also bring up this loop here again and just go with the failover scenario here again after we can't hit east. the client just keeps hitting west for the remainder of this order. Let's actually take a look at the restate cluster. What does the restate cluster tell us? If we look at restate CTL status, first of all, it tells us here's a dead node, node number zero, last seen a minute and 42 seconds ago. And we'll also see that because we have only two nodes left, we only have two partition processes attached to each partition, a leader and a follower between nodes one and nodes two. If we go to the log, we can actually see something interesting here that for some of the loglets, we got an epoch bump. You can also see this here. Some of them are an epoch 1 and epoch 2. An epoch bump here also is an indication that a leader of that particular partition moved away and a new leader took over, which means we're getting actually a new subloglet with a different ID. Good. Finally, let's actually try and bring everything back up again. and see how everything just recovers. So we're going to start US East 1 again, and let's actually hope that DNS is quick enough to pick this up. And we should see that after a while at least, once we start trying to hit US East 1, these requests go through again. So once this bottom script here, for example, is done with one iteration, it starts hitting US East, and we should actually see those continue again. Very well. a node in east actually came back and is processing our requests. That's great. We can see the same thing in the script above. Now here we're hitting west, west, or should be hitting east at some point again, and it should just work like we're bouncing between east and west. Here we go. We're also hitting east endpoints, and it works. Looking at the state of restate CTL here, we can see no longer dead nodes anymore, and we can see there's three partition processors again on every on every partition, a leader and two followers. And we're basically back to a functioning three-node setup. That's it.

A demo of Restate running in a replicated active/active setup across multiple AWS regions. The video shows an order processing application and demonstrates how Restate handles failures of a region and seamlessly continues in the remaining regions, without any loss of consistency.