Consistency and Coordination Patterns in Event-Driven Architecture, Emanuel Trandafir

Hello, welcome. I'm Emmanuel. I'm a Java developer from Bucharest, Romania. I'm also an author on Baildang. I'm trying to contribute a bit to open source to some projects. Outside of work, I'm trying to stay active, so I'm doing a few sports, rock climbing, tennis, triathlons. Maybe like 10 years ago, when I was doing this a bit more seriously, I had a favorite race here in Bulgaria. It was a 10-kilometer race in Veliko Ternovo. It was called Cross Tivailo. It was pretty tough because the route was very hilly and the competition was very tough. So I went there. I finished sixth. Then I went home, trained, returned back. Next year, I finished second. Then went back home, trained more. And then next year, well, next year, I learned Java. So I stopped running. So OK. So why architecture? Why have I chosen this topic for this day? Why event driven? Why all these patterns? Well, now that we generate a lot of code with AI, and we have all these nice tools, we don't spend so much time just typing code, just writing. So we can use this time somewhere else, right? And I think there are some interesting areas where we can use this time. Apart from teams meetings, I think Observability is an interesting one because we want this feedback loop, right? We want to see how the generated code is behaving. But today we'll discuss more about design and architecture. Simply because we generate a lot of code, we integrate it more often, and we need to stay on top of things and make sure that everything fits into our structure. And it's also because I'm from Bucharest, you know, from... Romania, we are in Balkans as well, I saw some really bad architecture so when I was 10 my parents took me to Constanza to the seaside and I saw this and because this building was really close to the city center you know they tried to decorate it for the Christmas fair so we had the Christmas fair they tried to decorate it to put some nice Christmas decorations you know just to make it look festive so they put some teddy bears and this was the result Now, I don't know if you worked with monolithic legacy old applications and you tried to put some nice new features on top, maybe reactive streams, I don't know, maybe AI now. Was this the result? Oh, for me it wasn't. So that's what we ended up with, yeah? Now, how did we end it up here? How do we have code that looks like this? Let's start with this bit of code. It's the only bit of code that I'll share. But we will assume we are working for this e-commerce application. And this is the Java method that is creating an order. And as we can see here, we are saving something to the orders table. And then we are also updating the products table. And at this point, we would like to make this transaction, to wrap this in a database transaction. So we want to be atomic here, to have these two operations atomic. Unfortunately, this method is doing more than just this. It is also publishing a message to our message broker. And in fact, this is a domain event. So it's a way for our order service to communicate to the rest of the system that, look, something happened within my bounded context, the context of orders, and the rest of the system needs to know about this new order that was created. Then it's also saving something to the local file system. It's also sending a post request to a different microservice, this loyalty service. And finally, it's also sending an email using this email service bin. Now, as you can see, this code is doing quite a few things, right? It's definitely not single responsibility principle, and Uncle Bob will be really upset with this. Yeah? Now, have you seen code like this or wrote code like this? What about your colleague? We are doing all sorts of side effects here. So the test is kind of trying to tell something that's weird here. And if we look back, so the problem, why I said we cannot really, or like it's tricky if we just mark the whole thing as a single transaction. Well, what if one step fails? What if sending the email fails? We will roll back the changes from the database. But what about the Kafka message? The Kafka message is already out, right? It's in the broker, maybe consumer applications already received it. Same for the file and same for the post request. Maybe this loyalty service is already processing that request. So we'll be in an inconsistent state. Transaction works for the database changes in this case, but not for all the other changes. So we have two types of problems here. One is this data consistency problem that we discussed. And there's also a coordination problem. So we have five different systems that we are trying to coordinate, and we are having some issues with this. And if we summarize this, our challenge was writing data to different systems. And this can involve changing something in the database, publishing a message, sending HTTP requests, and so on. There can be many examples of this. particular problem or challenge is also known as the dual write problem. And it's pretty common in the distributed systems. But this is the thing that we are trying to solve. So today we'll discuss different consistency and coordination patterns that will help us find a better way of doing this. And also it's dual write, I guess, if you have two systems, we have five. So I don't know how we can call that. Now, out of these side effects that we are doing. Which one would be the easiest one to undo if you want to undo it? So we sent an email, we sent a Kafka message, we were saving something in the file system, and we sent a post request. If we just want to deal with this side effect, like just deleting the file is the easiest one, right? We can just, this is very simplistic, but we can just catch some exception and just delete that file and We have some consistency again because something went wrong, we rolled back the data in the database, but we also deleted the file. So between these two, it's more or less like a consistent state again. Now, why can we do it here and we cannot do it with Kafka or with the message broker? Sorry, if Kafka slips, this should be just applying for any message broker. Well, we don't have this control over the broker, right? But... If we have control over the consumer application, maybe we can do something similar there. So the idea is that something went wrong. We just go and perform an operation that tries to undo what we've done. This is called a compensating operation. So let's imagine this case. The order service receives the place order requests. It's going to save the order into the database. And then it's publishing the order created domain event. Now the inventory service is listening to this, is doing its own bit of logic and something breaks here. Let's say we are out of stock. Now in this case we can publish another domain event message saying that look we have an issue here. We have the stock unavailable problem and order service can listen to this and perform some compensating operation. In this case it can cancel the order. So we are in a consistent state again between these two databases. Now, cancelling the order needs to be idempotent. So we should be able to retry this again and again until we succeed to cancel the order, right? So that's why we say we are eventually consistent here. It can be out of sync for a while, for a bit of time, but eventually we should be in a consistent state again. So this is good. It works. Now, here... Order service knows about inventory, and inventory service knows about order created, so they both kind of know about each other. And there's this almost like cyclic dependency here. So what if we can, can we try to avoid this, to break this cyclic dependency? Well, we can if we introduce a coordinator. So we have this e-commerce application, which is just coordinating this workflow. The e-commerce application can just tell the order service, look, you need to create an order. This time, this is no longer a domain event, right? It's just telling it to do something. It's a command. It's passive aggressive saying, you need to create an order. And then it goes to the inventory service, and it yet again sends a command to update the stock. If something fails here, The coordinator can go to the previous steps again and send another command to do this compensating operation, in this case canceling the order. And of course, we can have multiple components in this workflow. So there's another one there. And if something goes wrong here, of course, we go to both of the previous steps and undo them. So these two were like two flavors of the same pattern. And the essence of this pattern is just that we do something and then if something and we perform a local transaction and we move on with the flow and then if something goes wrong we just go back to all the previous steps and perform this compensating operations to be in a consistent state again and maybe do you know what's the name of this pattern? Yeah, somebody said saga so this is the saga pattern so the definition from the microservices book is just Saga is just a sequence of local transactions where each local transaction updates its own database and then we publish a message that triggers the next local transaction from the saga. And then if something fails, we just trigger this series of compensating operations. That was the saga, right? This helped us, you know. It's one way to tackle this distributed transaction and this dual-write problem that we saw earlier. Now, what were those two flavors of it? So this was one of them. So here, this pattern is named orchestration. And this coordinator is actually an orchestrator. It's like the agent orchestrator talks, but this is the real deal, the initial one. Now, what's interesting about this is that, first of all, it's really simple to implement Saga here. So the error handling is pretty simple. And also what I like here is that if you see, the order service doesn't know about inventory service and so on. Inventory service doesn't know about shipping service. So they are very decoupled. They don't know. They're very independent from each other. So when I saw this, I was like, really happy. I said, well, look, this is a system with no coupling, right? But it's not true, right? Because what we do is just to move all the coupling to the left-hand side to the orchestrator, because the orchestrator needs to know about all the components it orchestrates and hold all this logic about the business workflow that we have. And it's also harder to implement new components, because if we implement a new component here, a new microservice, let's say, stock replenishment service or something like this. It's not enough just to deploy this new component. It will do nothing if we just deploy it. We also need to go to the orchestrator, include this new component into a domain workflow and redeploy the orchestrator so there are a few more steps to do. So this was orchestration. The essence was that we have the central coordinator that tells the orchestrator components what to do. When to do it. Oh, it doesn't tell them how to do it. This is still responsibility of each component. And all these orchestrated components, they don't really talk to each other directly. So everything happens through the orchestrator. Okay, so this is the orchestration. Now, the alternative would be this. No central coordinator. And all the components are just listening to the main events, doing a bit of and publishing their own domain events. This allows the system to grow faster. This is choreography. So what's nice here is that it makes it much easier to implement new components. And because we no longer have the central coordinator, there's no bottleneck or no single point of failure. So it's a bit better from this sense. On the other hand, we still have this complex error handling. So if something goes wrong here, it's pretty hard to notify everybody what happened. And yeah, we have the cyclic dependencies that we talked about. So this was choreography. Services are loosely coupled. So they just communicate through domain events. And they decide by themselves what to do and when to do it. So we had all right. the main changes we are trying to solve. We discussed about Saga, which is a consistency pattern, helped us keep everything consistent. And then orchestration and choreography, they are two coordination patterns. So they allowed us to coordinate all these different systems. And now when I had this talk before, somebody asked me, like a friend of mine asked me, like, okay, Manuel, this is pretty cool. Thanks for that. But just tell me which one should I use on Monday? I don't know. orchestration, choreography, just tell me, Claude is also not telling me, so just say one hand, you know. And it's not that simple, right? So if this is our system, it really depends. Maybe we have one part of the system where we have a complex error handling and we want to be able to use orchestration there. And for the rest of the system, maybe we still want this evolutionary architecture where, you know, it's pretty easy to add new components. So we can use one, the other, a mixture of both. Then what if we spend too much time implementing sagas and dealing with all this inconsistency? Or in that case, maybe there is a sign we broke the system into two small pieces, pieces that are too small, and maybe we can just have a monolith. I don't mean this monolith, just a nice modular monolith. So if this is interesting to you, you can also... read about spring modulate. That allows you to keep one or it's a bit more tidy and nicely decoupled. So until now, we used the saga to keep everything consistent. But we always started with this assumption that we can send the message. And the message is, like, the consumer is listening to this and undoing something. What if our message doesn't even reach the broker? So this is my message trying to reach Kafka. So if this was our code, saving to the database, then trying to publish something, this is just a diagram of the same thing. What can we do here? So one option would be to try to wrap everything in a database transaction, again, like we discussed before. And this is a naive approach. It doesn't really work. But the reasoning is pretty simple. It's that, OK, I'm trying to save the database. Then I'm trying to send. And if sending itself fails, we'll have an exception and we'll roll back the transaction. So we are in a consistent state here. But it doesn't really work. One second. We actually tried to implement this with just wrapping everything in a transaction. So the issue we had was we were just creating this batch of promotional codes that the user, just some string values that customers can just put in their application and get some discount. So we created the batch of these codes. We then published the domain event that, OK, this batch of promotional codes is ready. Other applications started picking them and distribute them to the users. Something failed, so the insert into the database failed, because now it's a transaction, so the commit happens at the end, and everything was rolled back. So it was looking like this. Everything looks all right. We tried to commit, we failed, and we rolled back the data from the database. So there were no promotional codes in the database, but there were these promotional codes in the rest of the systems, and we were already distributing them to our customers. And it was really hard to debug because they look real, they look very like exactly the format we would use, but even we couldn't find them in any audit, it was pretty tricky to understand what happened. So just wrapping everything in a transaction doesn't really work or it's tricky. What if we stop here, no transaction, we just stop here. Because after all, just publishing the message itself It's a pretty simple operation, so maybe it doesn't fail. Well, this is actually at most once message delivery, right? So we can either send a message or we can fail to send it. We don't have any guarantee that the message will be sent. And in some cases, it works. Maybe it's just good enough, maybe. Maybe if we are sending some promotional emails or something like this, we don't want to overcomplicate things for some... tiny edge case. But in other systems it doesn't really work. So for example right now we are working in financial markets and we have a low number of transactions so not many so the throughput is not very large but the value of these transactions is very high so we cannot afford to lose any message not even in these edge cases. So at most times it's not good enough in this. in this case. Now if we stop here for a second, I want to tell you a small story. I told this to my father. My father said, son, you're old enough now, you need to know that Santa Claus is not real, Toot Fairy doesn't exist, and there's no such thing as exactly once message delivery. Because I wanted it exactly once. I wanted to be sure that. So I was really upset. At the Christmas fair, all the kids were getting presents from Santa. And I was just very upset. But guys, you need to remember I'm from Romania, so 20-25 years ago Christmas didn't look like this. I don't know if it was like this in Bulgaria, but for me this was the Christmas fair, as you can remember. And this was Santa. So in fact, you can see me there, a bit worried. But after all, I was pretty happy that the guy is not coming to our house to bring presents. But I was mostly concerned about Kafka. I was like, what do I do with this? So at most times, it's not good enough for this case. Exactly once, it doesn't work. So we need to find something else. And something else is just trying to have this at least once message delivery. And by the way, when I'm saying that exactly once, it... It's not possible. I mean this context of saving to database and then publishing the message, yeah? Because your message broker probably supports exactly once, but not in this context of the distributed flow, like also invoking the database as well. So we wanted to go to at least once. And we have a pattern for this. So maybe outbox pattern. Maybe you heard about it. Have you used it? Yeah? If you haven't used it, you probably used an email that uses it. Yeah? So when I'm trying to send an email to you, it first goes to a special place called the outbox. Then a separate process is just looking there, taking emails to be sent, and then it's sending them to the intended receiver. So it's breaking this into two different steps. So we'll implement exactly this. Instead of trying to do everything all together, we are breaking this into two pieces. First, just saving the order, later publishing the domain event. Then this outbox space is going to be a new table in the same database. It's very important to be in the same database because we want this to be atomic. So now everything is a single transaction. Another good thing about this is that now this part of the code, it doesn't keep the database transaction open for long. So it's just saving, inserting and then releasing the database connection. It doesn't keep it open for other I.O. that we were doing, like sending to the message broker or doing requests and things like this. So that's another thing. Then how do we connect this? We can either have some change data capture mechanism or just polling. We take the messages one by one, we publish them, and finally we update the outbox table, just marking the record as published. And it looks like we solved this dual write problem, right? Because if you look here, we are only talking to the database, so it's not the dual write anymore. However, on the other side, on the right side, we are still publishing the message and then trying to update or remove from the database, so it's still the dual write. Yeah, we just moved. the problem further away towards the boundary of our application, right? But this is good, actually, because it allows us to do the following thing. So it works like this. If we cannot publish something for any reason, the next time we pull this message, we'll send it at some point. The problem is that we can still publish the message, but fail to update or to remove it from the database. And this means that the next time when we are pulling the messages to be sent again, we'll end up with a duplicate message. I will send the same message again. Now, this is something we knew it can happen. This is why it's at least once. But it's easier to deduplicate the messages later on than to try to recover messages that were lost. And we can even do one more thing. We can help the consumer application. to understand that this is a duplicate. The second message is duplicate. How can we do this? Well, in the outbox, we can have a column which is called idempotency key. You can give it another name if you want. But this will be a unique ID and it will become a header to our message that we publish. Now, if we send duplicates, all of them will have the same idempotency key header. And this is a way for the consumer application to understand that, well, look, I already processed this message, so I can just discard the second one, and so on. Additionally, we can add all sorts of things that may come in handy, like an observed at header. So this is more that in case we want to reorder messages later on in a given time window, but it's a bit more complex. So we can have all of this. It brings a bit of extra latency when it comes to publishing these domain events. On the other hand, the business side of things here, it's going to be faster because it's not dealing with other I.O. So, yeah. It can produce duplicates, as we know, but it guarantees the delivery. Now, and also, it brings quite a bit of complexity, right? table, we need to do this polling, we need to check quite a few things. So when is it worth using this? So we can use it when our operation, we can use this when we cannot really afford to lose messages. So it's very important, like the case I described before, especially if fault tolerance or like resilience is very important. Yeah, one example would be and financial markets where you don't really want to lose any message. We can keep using it if at least once is good enough. So maybe in some cases where you are having a, maybe you're reading from a sensor, let's say, you have quite a few messages and just losing one doesn't really impact things because the others are coming very fast so we can just drop that one without worrying too much about it. And about the boiler, plate code and all the complexity if you want to use something to help you. Spring modulate has support for the outbox pattern. And yeah, you can use something for the change data capture as well. That should help managing this. So we started with the dual write. Yeah. So the issue that we had, like the challenge that we encountered when we were trying to write to different systems. We discussed the saga. Basically it was about trying to perform a compensating operation when something goes wrong. We just go back and undo what we've done. Then for coordinating all these systems, we discuss orchestration and choreography, two different ways of coordinating them. And we discuss the outbox to make sure that our message actually reaches the broker. And by the way, you can use outbox with the other side effects as well. So you want to make sure a send request. is sent successfully or something like this. So it works with multiple different implementations, I guess. OK. We discussed, we said that we are guaranteed to publish the message, but we may have duplicates. What do we do with those? So this is the consumer application receiving a bunch of duplicates because we implemented the outbox. OK. So we can do something pretty similar. This is called the inbox, yeah? And it's pretty similar because we broke this into two different pieces as well. We have, we first take the incoming message, we save it to the inbox table in our database, then we pull these unprocessed messages from the inbox, we process them one by one, and finally we mark them as published. Now, if you remember about those headers, Now, having this inbox is really convenient because we can have a column in the inbox to store the idempotency key. And we can put the uniqueness constraint on this column. This means that we delegate to the database the responsibility of deduplicating these incoming messages. And of course, we need to make sure that if we fail with the data constraint exception because of these duplicates, We just catch this exception and we acknowledge the message. You don't want to get stuck there. Then obviously we can differentiate retriable and non-retriable exceptions. So a retriable exception will leave the message in the inbox so it can be pulled again. A non-retriable one will either mark it as already processed or we'll just remove it from there. We can try to use the observed at header to reorder messages within a given time window. But as I said, this can be a bit more tricky. So yeah, this one as well adds a bit of extra latency because we have this new table that we go through and we pull and we do all of that. Now if we drop all of this again and think a bit where it makes sense to use it. So basically we can use the inbox to implement idempotent consumers, right? Consumers that can easily deal with duplicate messages. So if our operation is not naturally idempotent, we can use this to help, to make it idempotent. Now, if we feel like it brings too much complexity again, new table, new polling, we can avoid it and we can use a certain combination of features. So we can either have operations that are naturally idempotent, or we can change our design to make them like this. So for example, just inserting something if it doesn't already exist, and using a business field or column to decide this. Maybe the client can send the unique UID, and we can have this in the payload. We can have retryable topics again to be able to retry messages without blocking the main topic or queue. And we can have a dead letter queue just for audit and to be able to manually inspect the messages that were lost or we exhausted all the retries for them. So I think these are pretty much the things that I wanted to discuss. So we started with dual write, Saga, orchestration, choreography, outbox, and then inbox. In the software architecture, the HUD parts, in the beginning of the book, the authors are even talking about some decisions that you need to take when you want to connect to the different components. You create this link between them. And they are saying there are three axes for this decision. You have the consistency, so it can be either atomic or eventually consistent. We were like in the eventually, our patterns were in this eventually consistent part of the 3D space. We had the coordination, which we already discussed about choreography and orchestration, and the communication style, which can be either synchronous or asynchronous. So everything that we discussed, all these patterns are somewhere in this three-dimensional space. We started from the dual-write. This had to do with updating different systems. Then Saga helped us keep this consistent by having this way of communicating that something went wrong and performing some compensating operations. And this is the Saga in an orchestrated environment. Orchestrator had the central coordinator that coordinates everything and... It can send commands to undo previous steps. And choreography, it's very convenient because it allows us to grow our system pretty fast. It has this evolutionary architecture. And we said we can use one, the other, or even a mixture of both, or even none, if it doesn't make sense. Then if we have problems with losing messages and we want this eventual consistency and guarantee message delivery, you can use the outbox pattern. And in order to deal with the duplicates and implement at the pot and consumer, so you can use the inbox pattern. And I think that's pretty much it from my side. This is a QR for feedback if you want to give me some. And I will post the slides on LinkedIn a bit later today. And we have quite a few minutes for questions if you have some.

AI can help, but we’re still the ones accountable for architectural decisions that can lead to lost messages, duplicates, and other data inconsistencies. In this session, we’ll tackle the dual-write problem, explore the Saga pattern, and compare orchestration vs. choreography strategies for managing distributed workflows. We’ll also dive into the Transactional Outbox and Inbox patterns, showing how we can achieve at-least-once delivery and build idempotent consumers that make our event-driven microservices more resilient, consistent, and reliable. Target Audience - Backend engineers building event-driven microservices with Spring and dealing with data consistency, messaging, and distributed workflows - Junior and mid-level developers who want to understand the challenges of keeping distributed systems in sync - Architects and tech leads responsible for designing reliable distributed systems Key Takeaways - How can we handle common challenges such as dual writes, lost, out-of-order, and duplicate messages - How to choose between orchestration, choreography, both, or neither - When and how to apply Sagas, Transactional Outbox, and Inbox patterns, including their trade-offs