Event Sourcing in production — Borjan Tchakaloff

Hi everybody, I'm Borjan. So the mic is alright, yes? So I'm here to talk to you about running an event sourcing system in production. I've been working in Python for about five years now, I guess, and software for 15 years or so. We were using event sourcing system at my previous job for a few years, and I'm trying to compile some experiences. and hopefully some tips for you today. So in a nutshell, once I'm done talking here, I would like you to keep a positive idea about event sourcing. Maybe think about it and consider it for your next project or your next big refractor. So I will talk about some roadblocks and patterns we use to solve them, common things that I think you will encounter when you start in event sourcing. We have a very nice library in Python to handle event sourcing, which is called event sourcing. Look it up on PyPI. It's flexible and it provides a lot of really solid building blocks. So I'm going to use the library today in my code examples to illustrate a bit what I'm talking about. It's not going to be a tutorial, so I'm going to make some simplification on the library API itself. And I'm also hopefully not too much, but maybe muddy a bit the water with definitions and concepts. I'm trying to be a bit more pragmatic and I don't have too much time. So the event sourcing and related concepts I'm going to quickly touch on together with good examples to show you a bit what it can look like in Python. After that, I will talk about evolution, projection, domain and runtime. names and concepts will also be explained along the way. So first of all, my example today is not going to be a cargo shipping or bank transactions or shopping cart, which is usually what you see with the management design and event sourcing. I'm using something a bit more simple, a bit more simple. It's still a contrast example because, well, we don't have too much time today. Imagine you want to meet with your friends and you just developed basically a gathering service. So just for your group of non-people. So the service is quite simple. You don't have a lot of concurrent users. And that also means that your domain, the way you are working, you don't have to handle too many concepts. And I will show you a bit later what I mean by that. So it's in the title, What's Up With Event Sourcing? Simply put, it's about capturing the state changes into a sequence of events rather than just persisting the current state. These events are the one and only source of truth. Everything else is derived from them. So in comparison, the traditional approach persists the current states, which gets overwritten when you make a change to it. In comparison, event sourcing offers a complete history of all the changes and you can you will build the state from it. If it's a bit confusing now hopefully with the examples you will see what I mean. We have been doing this kind of things for about 5,000 years already. We found some clay tablets in from Mesopotamia from 3000 BC and basically they were already keeping ledgers of market transactions and A ledger is basically a recording of transactions. So those are the facts that happen. It's not the tally, right? It's not who owes what right now, what's the total. That's a projection. You use all the transactions, all the events to eventually count where you stand in your tally between basically two sellers or a seller and a buyer on your market. Greg Young, whom I quoted here. is well-known advocate of event sourcing and demand-driven design. And I think nowadays most people using event sourcing really blend the two together, at least the technical aspects of it, which is what I'm also doing today. So demand-driven design, what is it? It's a bit more recent, only about 20 years old now. It was described by Eric Evans. And it's... It's basically about sharing knowledge, having a common understanding between the operators, so the users, the practitioners of the domain, and the modelers, so us developers, the technical folks. It's a methodology, it's patterns, it is many things. But here, in a nutshell, the technical aspects of it, you have basically the domain which is a bubble, the specific environment the business operates in. And today I'm going to equate a domain to a namespace, so a module in Python. And then you have aggregates, which are the smallest addressable units in your system. They enforce a boundary. So nobody can ever reach within an aggregate to state the change. Everybody has to go through the aggregate, which means that we aggregate. is used to maintain the invariance. So any constraints is being enforced by the aggregate. And aggregates, it's quite important, are not about relationships. They're not about hierarchy. So don't think about it as a class diagram of your system. It's more about the cases that exist in the domain and how best to support those rules, right? All the exceptions and all the conditions. You know the business rules. From there, you can create the aggregate and the structure is just enough to support that. You don't need to go overboard. So what do we get when we mix event sourcing and domain-driven design? I'm going to quickly show you with the example of the event sourcing library. So that's just a high-level view of the core domain classes, the aggregate and the domain events. And on the left, those two classes are basically meant to be derived. Those are just parent classes, right, with metadata. And on the right, you can see how I'm using it. So what can happen to a meeting, basically? What are the facts that we want to record? What are the events? And so a meeting can be created by somebody about a topic. It has a schedule. It can be edited. It can be rescheduled, canceled, and so on and so forth. How can we use these events now briefly? Also still to illustrate the point. So there are a few ways to weave the logic and the event together. And here I went for a two-way binding with the event decorator over the done-doinit method. And this basically makes it so that when we instantiate a meeting aggregate, a new event created in this case is created and queued for us. but it's quality of life decorator basically. We can do it with from first principles, but this is just a commodity of thought by the library. And from the other side, when we have like a sequence of events and we want to reconstruct the upgrade state from them, this binding also makes it so that when we see the meeting the created events. the Dundoynit method is called for us with the same attributes. So the attribute of the event matches the attributes from the Dundoynit method here. Something else that we can see is the divergence here between the attributes from the aggregate itself and the attributes that we have on the events. So if you don't really see it, I'm going to tell you. So we have For instance, the edited or canceled events have an additional reason attribute, right? Which makes sense when you think about the facts that happened to the meeting. But they did or canceled. And if you want to notify your participants, you will want to know why. But for the purpose of this aggregate, I am not storing reason or the actual meeting owner even. Because this aggregate... the method it has to edit or to cancel, they're about the invariance, they're about enforcing that the topic is not empty, right? It's not about who created it. I have no invariance in this one in this case, so I don't need to care about it at the aggregate level. But I do want to care about it in general. But it's not here that I'm going to care about it. I will explain a bit further where. The point is that I'm separating the commands and the queries. And this is the command query responsibility segregation pattern is basically about keeping the commands and the queries separated, as the name implies, and the data structures that they use. So the idea is when you want to make a change, you are focused, targeted. You don't need to know about the whole world, right? You just have a very simple structure to initiative commands. And on the other side, when you query your system, you will likely want to have more information available. You want a broader view. So hence the difference between the right models and the read models. And since the aggregate is this structure to enforce our invariance, it's basically all right models. And I'm keeping the RIN models aside, but we can also have them in the aggregate. But I will show you an example which I've been using in production. It's basically having RIN models projected to a Django model. First off, a quick look at how we... a bit of infrastructure with the EventSourcing library before looking at how we can get the RIN models. So the application is the programmatic entry point to a domain. It basically holds the public commands in the domain and it provides a means to persist the aggregate events. Yeah, and we can basically persist one or more aggregates at once atomically. That's the point of application. It is the unit of work. A process application is just another application. but that knows how to react to other domain events. So typically this is where you will have local caches about aggregates from other domains or that you will trigger more comments based on the fact that happens prior. More briefly, the system is just the way to detail the graph. So which application is subscribed to which application so that we know where to send the domain events basically and the runner is simply the executor which pushes events around in the system. So an example of an application here with the meetings basically I guess I will go quickly about it. You can see that so in the create method we are basically creating a new meeting aggregate And as I explained earlier, we have a binding, so this also queues up an event for us. So when we save the meeting in the application, then what happens behind the scene is the application collects all the pending events from the aggregate and saves them. Because what we want to keep is the events in the end, not the aggregate per se. Which is illustrated in the second method, edit, where we look for the aggregate. And then we can actually execute a command on it. And here, the self repository get meeting ID is basically hydrating the aggregate from the events that are stored. Well, new events stored. So they are applied in sequence. And the state of aggregate is rebuilt that way. So this is kind of glue code, basically, to not expose the... the persistence mechanism to the outside world. And I have other examples in my extras. I'm not going to show that in the main run of the talk, but the idea of save is to be atomic. So we could potentially save multiple aggregates together or fail together. Yeah, so the other methods are left here as stubs. So we could also add a new method here. to basically run a query and rebuild aggregate and then expose some fields we want from it. But as I said, I prefer to have a dedicated structure for that. And I will actually show you here an example of two of the structures. So a pretty much one-to-one structure would be the meeting info. It's very close to the aggregate in terms of structure. We know about the offer of the meeting, we know about the topic, about the schedule of the participants. So basically this is the kind of structure you want to use when you list the meetings that are coming up. If you expose that to your web application, that's your list operation. And another kind of read model or projection we can make from the same events is statistics. This is a very common approach like reports, statistics. The same data is basically viewed from a different perspective. And here it's I don't know if you were here earlier for the aggregates and Django view models, but that's that. It's basically an aggregation of the different meetings. And here just some simple stats with number of meetings and number of participants. So that's just to give you an idea that the how I view my data, I keep it separated. This is the real model, it's not the right model. I don't need to know about the statistics to schedule a new meeting basically. All right, so let's go a bit further and now we are done with the concepts and the illustration. Let's go into use case territory a bit more. So I will begin with the projections and first of the simplest case. just when we need a new projection and then a quick look at handling a projection that needs to change. So say that you have a domain which events you're interested in. Hint, hint. I am interested in the meetings events. So I will write a projection for that. You can also have multiple domains you're interested in. That's also a strength of projection. It doesn't have to map to your aggregates. So we will basically subscribe to the domain events with a process application and in the callback, in the policy method, we will then have logic based on the type of the event that we see. And I would suggest to when you are going to production to basically fail loudly if you encounter an event you don't know about. Don't be lenient. Don't have a default case in your match statement. The idea is I think it's better to fail and then fix than to silently swallow or ignore the unknown event because then you will get to a broken state, basically in your projection and you might not notice it. Obviously that's you want a good test coverage to catch this before going to production. It's better than having 500. But the point about the read models is that if you have many queries, so queries in the general sense here, and they don't really intersect, that's a good use case for having different read models, right? Just like when you have... It's basically about normalization and denormalization. And you can choose which one makes more sense based on your use case. So a quick example. here on how you would do it with a Django read model. So you would have a new dedicated process application that is basically subscribed to the meetings application, which saves the events of the meeting as aggregate. And the idea here is when you basically dispatch on the domain event type here. And for instance, for the created event, then you create a new entry in your Django model. When you have an edited event, then you update this entry based on the identifier. And when you have a canceled event, then you can just delete the entry. Because this is just a state you can rebuild. It's transient. You don't have to maintain the history here. It's just really a flat state. And on the last line, you would... Basically, that's how you would say that meeting info project also this projection application subscribed to the meetings application earlier and you can have many subscriptions in here. So that's the initial implementation of your projection. You write that, you run it, you're happy to have the read models there and then something happens. Requirements change or the read models. of the domain events change and you need to update your real models or your production. And once you've done that, you have to replay the whole history to basically catch up the production with your domain events. So there are two aspects to dealing with that. On the real side and on the productive side. So on the logic, I have to hurry up a bit. So the easiest would be to basically delete all the entries, update your schema, and then keep your application offline while you reproject. And that's what I recommend. If you can do that, it's the simplest. Just do it. Another approach would be to update in place your read models. But then you have a few more gnarly things, like you need to migrate also your read model schema itself, so you have to handle it. conflicts there because you have existing data you have to migrate. Plus, you will get new data on top. And if you happen to query this data before the projection has catch up with the domain events, then you are in an in-between state. So eventual consistency kicks in and basically you are lagging behind. Another easier model to reason about, create a new projection. a new model, a new table, a new database and I would say switch to that maybe using more kind of a blue-green delivery mechanism. So you switch to new entries as the old entries become obsolete once you are catch up basically on the new entries. But this requires a bit more infrastructure that is not built in the library. On the logic side, well again we have the simplest conceptually write a new projector because then you don't have to worry about its own state. It will just basically execute the whole history again and then you will have your fresh read models. When you have Django you can pretty easily, I mean if you're using Django also to maintain the state of your projectors like you can do with library, then you can pretty easily reset the state there and on next startup your basically projector will replay the history again that's a nice trick um yeah so if you can keep your system offline and replay uh well it's there it's easier and then you have it you have your consistent system on restart and yes okay so let's shift gears um yeah so let's talk bit about the things that can go nasty like domain evolution and I want to talk about two things domain extensions so adding a new attribute and in-vanish restriction when you want to for instance add a unique constraint after the fact so I'm going to use the example of the meetings and imagine that you are growing the circle of users a bit more and then now you need to also care about the location because it's not only one city it's only one place there are more people so You have two ways of doing it. You can either upcast existing events with new attributes or create a new event specifically for this new attribute. In the first case, you basically are deriving more information from existing events, which is fine, as the new consumers with new projectors will basically get the data that has been upcasted. But the projectors that are already up to date, that have seen the data, won't see it. So you are responsible to trigger an update for them. So that's basically fine when you don't have consumers or when it's only projectors that you can just reset and replay. But it shows its limit pretty quickly. semantically speaking, because then there is no trace of what you're doing in the persistence. The upcast happens in memory. So if you basically were to only keep your event store, then you wouldn't have this information in there. So just keep it to metadata and don't choose that on core attributes. You could also create a new event. In this case, that's location changed event. And that basically keeps the backwards compatibility and just bridges the gap with existing events. Everybody will get this event as soon as they handle it and stop crashing on it. And the best thing is then your event sequence in your event store has the semantics of you can see that the location has changed. It's in there and you're describing basically what happened in your history more. concretely. So in this example of location, it makes more sense because it's a core attribute to our domain. So that's what we want to do. But of course, when we start, no aggregate have this information, right? It's undefined for all the aggregates. So how do you fill the gaps? Well, you need a kind of a workflow to do this. So either you are able to derive the value from within the aggregate. from the current state, or ever you need like an external input, right? You need to basically get the data from the outside world. So quickly, the point here is that you need to know about your aggregates so that you can loop over them basically either from within the application or from the outside so that you can then ask them to derive the location, right? You need to hook in. basically your control loop. So in this case, that's the example is you just ask for a specific meeting to get this location. Just quickly passing over this. The other idea would be to then have a multi-step approach, so like a long-lived process in which you would basically keep track of all the aggregates and which ones are missing the value. And once you are up. up to date with all the aggregates here, you can ask your tracker, okay, so now that you know, please send a notification to all the owners of the meetings that miss the value. Right. Just be careful because since it is an downstream consumer, you are potentially lagging a bit behind as well. Right. Just move it. So the second part of this domain evolution would be introducing an attribute unicity after the fact. And obviously if you don't have duplicates, then you don't have problems. You can just introduce a constraint and you're done. If not, then you need to migrate your data somehow. And I want to stress out that this is not a technical problem. I've just shown you an example of how to solve this with the mediatorial approach. When you track each issue and then ask people to solve it for you. it's really you have to decide from a business perspective what do you do with duplicates. So the approach I just shown you could have also could merge all the aggregates together the ones that have duplicate value or you could use a temporal approach and basically only keep for instance the last one that was updated and discarding all the others right it's again this is really a decision you have to take based on your domain. It's all valid, technically speaking. That's a quick example of the temporal tracker, but it's a bit lengthy on the screen. I don't have more time, so I'll just skip over it. Again, you keep track of things and you just keep track of the last one that was updated. And in the end, you can basically have the list of the winners. And the final chapter... The final chapter is about synchronous systems and I will just keep the last slide which summarizes the different approaches and the winner in this case is once you start to have traffic and you want to handle concurrent requests go for a hybrid approach which is basically have more workers but single threaded runners. The slides will be available online and I'm available for more questions. And I guess that's actually the end of my talk. Thank you.

[EuroPython 2024 — South Hall 2A on 2024-07-10]


Event Sourcing in production by Borjan Tchakaloff

https://ep2024.europython.eu/session/event-sourcing-in-production


Event Sourcing (ES) is a powerful concept that lets you adapt your business logic without losing data and past states.

Whether your domain understanding changes or new requirements land on your lap, there is a path forward.

Join us as we talk about some real-world tactics we relied on to manage Event Sourcing in production.

We will accumulate a handful of patterns throughout the talk that will hopefully help you avoid pitfalls and bottlenecks.


Our use cases build on the `eventsourcing` library, a mature and well-rounded Python library that deserves more attention.

We will tackle the three key aspects of a successful event-sourced application: evolution, projection, and runtime.


Software does not run in a vacuum, models need to change and evolve to reflect the world they live in.

ES records the evolution of how we abstract our domain, how we see things.

Eventually these abstractions can become clumsy or simply inappropriate.

We can deal with that without breaking stride (losing data).


ES also gives us the ability to revisit our perspective and change how we present the application state — by creating new projections and replaying the history.

We will look at how it offers a cheap way to support optimal read-models, which we can can tweak and rebuild in the blink of an eye.


Finally, we will present how such a system actually runs in a typical web application.

Whether in the request loop (synchronous), or through eventual consistency (asynchronous).

As a single process, or distributed for parallel processing.


---


This talk assumes some familiarity with _Event Sourcing_ and its friends _Domain Driven Design_ (DDD) and _Command Query Responsibility Seggregation_ (CQRS).



---

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License: https://creativecommons.org/licenses/by-nc-sa/4.0/