Kafka: Outbox Pattern (Part1) - Polling Strategy

Welcome back and today I will show you outbox pattern in the event driven architecture very very important it has been asked a lot in interviews and I will also tell you what generally engineer tried to answer first okay but those answers are generally overruled because there are some gap in those and what's the correct answer okay so before we go deep into this outbox pattern, we first need to understand the problem which exists. Okay. So, if you see this, this is the problem statement which interval will provide you. Very, very simple method of a create order when this endpoint got hit. What it does it? It first tried to save in the DB. Okay. So, the order which has been received, we are trying to store in DB. and in the step two we are also publishing an event the most common use case in industry we have to store in db and also we have to publish an event nothing special in this okay so these two tasks we have to do in this method so interviewer will say what is the problem in this so there is one major flaw in this one So we are writing two separate independent system. So this is one independent system DB and this is another independent system which is Kafka. Okay, so we are trying to write into two independent system. Okay, so we can get program like DB get succeeded but Kafka fails. For example, your DB order repository dot save. this got succeeded this first got succeeded you are it is stored in the DB successfully but when you are trying to publish to the Kafka that got failed this could result in that the order exists in DB yes but no even published so maybe the downstream components like payment service inventory service which rely on this event they will not be able to update their system so it could result into data in case inconsistency similarly What we can have is Kafka got succeed but DB got fail. It's possible right? If Kafka is able to publish an event but DB we don't have. So what would happen is all the downstream like payment, inventory, they will update their system but order doesn't even exist in the DB. So this is like more impact than use case one. So This is the problem statement. So problem statement would be given to you only this, that what is the problem in this, then we have to say that, okay, this two issue can come. And then, actually the name of this problem is dual write problem. So once we have identified this problem, the interviewer will say that how you will solve it. Okay. So I will tell you one, two common answers, generally given by engineers, but it get overruled. Because these answers generally have some gap. The first answer, we will use at the rate transactional plus manual exception throw. What this solution does, so we have put at the rate transactional for DB. So what would happen is, so in step 1 we are writing into the DB. If this succeeded, we are moving it to the next step. Now the Kafka part we have put into try catch block. If Kafka got succeeded, well and good. DB succeeded Kafka succeeded good but if there is any exception happens in Kafka it will catch it and throw and because of this throw exception since it is put under at the red transaction it will roll back the DB also so very high level this solution looks good right so DB got succeeded if Kafka got succeeded well and good DB fails ultimately it would be rolled back so good DB succeeded Kafka fail so it will throw exception and DB will also get rollback So sounds good, right? But there is a gap see this one DB success well and good Kafka template sent message sent to the Kafka broker messages sent to the Kafka broker, okay? But there is some gap in the acknowledgement received by the you can say that by the broker or the acknowledgement message has been lost because of network issue. Now what you will, what your application as a producer application will think that hey, the message has not been published, but actually it is published. But the acknowledge packet has been lost in the network. You can say that because of network issue. So you will say that, okay, Kafka published got failed. You will roll back it. And when you roll back it, When you throw exception, you will roll back the DB also, but your Kafka even got published. Okay. So the issue is that order does not exit in DB, but Kafka even published. You got it. You got the issue. At the first glance, everything looks good, but there is still a gap. So that's why this solution generally get overruled. The second solution, which generally proposes two-phase commit, that we will have a coordinator. We will have a one prepare phase one for DB one for Kafka both will participate and Like normal two-phase commit protocol, right? So but the participants are DB and Kafka So they will first say that yes, I am ready now in the phase two They will come in and after that they will come in but the problem is this looks Working solution, but the problem is Kafka do not even support two-phase protocol two-phase commit protocol, okay So two phase commit to work all parties need to understand this protocol and Kafka does not support 2PC. So there is no use to discuss the pros and cons of this that okay what would happen this the thing is Kafka do not understand this prepare, commit and all. So you can't add Kafka as one of the participant itself. So this two things are overruled. The solution which currently widely used in industry is outbox pattern. So idea is don't write directly to Kafka. Write to DB first and then a separate process read the DB and publish to Kafka. Okay, so now see this. In this one, the first step is same. I am writing to order. I have got the order. So, order repository to save order, same. Whatever we were doing before also, the same. Okay. Now, in the second one, if you see, I am not publishing to Kafka. I have created a new table, Outbox table. Okay. And I am publishing the event details into that one. So, it's still a DB operation only. So, now they can participate in one transaction. If any of this fail, we can roll back. successfully okay so at this point of time kafka is not involved only db operation so in this outbox pattern a new table we you can give any name but generally standard is you can give outbox table outbox name or whatever the name you want to give so a new table say outbox table is introduced in that one what we need to store the event details whatever we have to publish So that we store into this. And this two node DB operation can be part of one transaction. And now later on, a separate process will read from this table and publish to Kafka. Again, lot of interview questions can be come to this, like what the separate process? We will come to this one. So there are two, Polar and CDC. Okay, I will tell you both. But at a high level you got it what the outbox pattern idea is that we will create a new table, new DB table and that DB table store the event details and a separate process read from this table and then publish to Kafka. Okay, so now this become overall compliant with what we wanted. So what is the schema of this outbox table? It's a normal regular DB table and you can like change it according to your need. You can give any name you want. okay so but this is generally high level we are what everyone use a unique id okay aggregate type order generally this is used for determining the topic okay which topic it will go even type is order created order deleted generally we don't use it like but still if your business logic has to know about like okay uh order But what happened to the order? Order created, order deleted, or something you have to send into the header, or you have some business logic on top of it. So at least it will give you more context. What happened to the order? Event type. Okay. And then aggregate ID. You can consider that this is act as a key. Okay. So that, this is very important. So that all the orders, let's say all the order, which belongs to customer ID, let's say key is K1. So all the order for this key should go to one partition. So that order get maintained. Let's say first order created, then order cancelled. Then, so let's say this two, order created, order cancelled. So there is a proper ordering. So they both should have same key, key one, key one. When they have a same key, then only they would be goes to the same partition. And when they go to the same partition, then the order would be maintained. First order created, then order cancelled. Got it? So aggregate ID you can say that used for key. Then your normal payload which you want to publish created at is published. So this is optional. I will tell you its use. Okay. So but this is something is customizable depending upon your business need. But this is the high level. We can determine the topic name. We can determine the key. We can determine the payload. Okay. That is the same thing I have explained it here. And isPublished is optional. I will tell you when it is used, when it is not used. So Outbox table is consumed in two ways. I told you that Outbox table, a separate process will read the data. Again, there are two ways this Outbox table data is consumed. One is Polar or you can say that polling. Another is CDC, Change Data Capture. Very important. Okay. So isPublished is used in polling. But in CDC it does not used. We'll see that one. So let's start first polling and then later I will also show you the CDC mechanism also. If you have gone through the Spring Boot playlist and in that one I have covered JPA. So in that one you will get to know how to create entity, repository, how to connect with the DB. So this is the same thing I am doing it here. Nothing is special. So this is just one controller. the request comes to this one. Then I'm calling this order service. Okay. And here I have at the rate transactional. Now we know that we have to do two things. First thing is we have to store into the order DB. And second, we have to store in the outbox DB. So for this two tables, we have entity, order entity, outbox entity. Again, please check JPA topics. If you have any doubt, what is entity or how to create repository? Then for this two entities I have order repository and Outbox repository. Now I have dependency here and I am just using it. OrderRepository.saveOrder and then OutboxRepository.saveOutboxEvent. So, OutboxEvent I have created aggregate type, aggregate ID, event type and the payload. So, very very simple. These all are like just helps in creating the tables and connecting with the DB. And this is the application.properties. If you have gone through the JPA part, you know that you can connect with. So in the JPA part, I have connected with H2DB, but the same way you can connect with MySQL, Postgres, anything. So you need to provide the data source, username, password, and the driver class. So with the same way, we can connect to Postgres, MySQL and any DB which you want but this is the very basic thing we have to write. So now what we have done till now is in the outbox table we have the data. Now we have to write the polar. The polar should poll the outbox table. So very simple way I have written one polar. You can say that it is just a scheduler. Okay. I have also covered the scheduler. How you can write it. So, this is one scheduler. I have annotated with at the red schedule. At what time it should run is that in this one I am controlling. Every two seconds this will run. Okay. So, every two seconds it will run. What I am doing is in the outbox repository, I am finding the unpublished event. And the batch size is 10. Here I have defined it in the application.properties. So, in the outbox repository, I have this method. Select testers from outbox event table. Okay, outbox event in the entity. This is the table name. Where published equals to false means it is not yet published. And order by created at so that order is maintained. Limit batch size. So, I am getting 10 record at a time ordered by created at. Okay. If there is none, return it. But once you get it, we have to publish it. Okay, so that's what I am just doing. Kafka template dot send. You have a topic, aggregate ID key, payload. Okay, once you have able to publish to the Kafka, set publish to true that yes, now you are publishing so that next one, this same record should not be paid. And when you publish the ad is time, just save it. Outbox repository dot save, this event. In catch, why I am breaking is, let's say I need to follow an order. Let's say three events for this key. Now if one event got failed, so it goes to the catch. If I do not break it, it will proceed, two and three. Then the order will break. So if one fails, then you should not proceed with two and three. That's why I have put break. So you can say that this is just one basic polar. And it's very simple to write it. So you understood the polling technique? But this polling has so many disadvantages and that's where the interview will say that okay Why do you have something better than polling? But first we need to understand the disadvantage of polling. Disadvantage of polling is multi-instance handling So let's say you have three application three instances instance one instance two instance three so means three polar so this is your polar one polar two polar three all three will run together. So you need to maintain that, okay, they should not pick the same row. Okay. And if they pick different, different rows, then they should not break the order. Let's say order all belongs to one key, one, two, and key one here, three. This can break the order. So if it process three, one, and then one and two later. So there are many problems comes. So we have to have like shed lock technique like concept we have to build out. So we have to handle the multi-instance handling. Then a polling gap like higher I am running the scheduler every two second delay. So even though event is created at 10 the next poll will run at 10 to itself. So there is always a delay depending upon your what time you are polling. Okay interval constant db load. Now let's say that there is no event at all but you have to continuously pull your db okay so means you are consistently putting a load on your db hey is there any event is there any event is there any event even though there is no event at all you have also need to take care of the db cleanup so since you are maintaining a table outbox even table it need to be cleaned up regularly as its size will grow rapidly duplicate processing what happen is sometime let's say you have a you have read one row you have publish also but you fail to update to published published to true that got failed so in the next run again this row would be picked up and you will publish it again so there this is very common that you will publish the same event multiple times so your consumer need to properly handle the idepotency so there are many challenges with the polling And that's where the change data capture comes into the picture. Very, very important. So in the next part, I will tell you this one.

Join this channel to get access to perks:
https://www.youtube.com/channel/UCDJ2HAZ_hW-DMJj_U0zN38w/join

Outbox Pattern Part2: https://youtu.be/OQRv228F9Dc