Are you done yet? Mastering long running processes in modern architectures

Yeah, again, thank you, Bina. Thanks for having me. Good morning, everybody, or good evening, good night, depending on where you join the live stream. Right, so long-running processes. In order to get started, let's talk about food. And I always like metaphors, and I like making kind of comparisons to real-life situations to also discuss certain, let's say, tech problems, because for me, that makes it always more tangible. and probably also get hungry by looking at those pictures. But anyhow, if you order pizza, let's assume you want to order a pizza now, then you have multiple options to do, though. Actually, if you live in a very small town like I do, the only option you have is you have to call your pizza place, which I'm pretty sure you cannot do at 10 a.m., which is at my place at the moment. But that's a phone call. right so you pick up the phone you the number you wait for the other party to pick up you give the auditor and so on so forth you know how phone call works um and looking at like the technical analogy that's um what's called synchronous blocking communication it's synchronous i get a response directly it's blocking me from doing anything else right um so synchronous blocking communication It has a feedback loop built in, so I directly get some feedback from my pizza plates if they can do the order, if they're busy, how long they think it will take, and so on and so forth. But I'm also temporarily coupled to the availability of the pizza plates. If they're not open, if they're currently busy answering the phone, if they're currently at the toilet and can't pick up the phone or whatever, if they're not ready, um i can't phone them i i have to do it again we try and so on so forth right so synchronous blocking communication i know very basics we start with the basics here today um but we get into more interesting stuff pretty pretty soon the other option you would have probably is sending an email you could send an email to to your pizza place um saying okay i want to have this pizza The difference now is that we have a queue in between the email inbox basically from the pizza place. So it's asynchronous and non-blocking, right? There's no temporal coupling. Even if the other side of the equation is currently busy, I can still send my email. So I'm done. I'm not blocked waiting for a thing, right? Because I rely on the queue actually delivering the email. How does it make you feel sending an email to your local pizza place? I'm not sure about you. For me, I would get very nervous, actually, if I really get a pizza. So I might actually pick up the phone asking if my email went through. But one way of avoiding that, and it's not the asynchronous non-blocking communication, which is the problem. The problem is I don't get any feedback. So you, of course, can still have a feedback loop so they could answer the email saying, OK, we got it, right? um and again that could be asynchronous they sent you email so they are not tied to my availability and so on and forth right um that's the very basics we need in order to discuss certain scenarios later on so synchronous blocking communication phone call asynchronous non-blocking communication makes sense i guess and of course i know most of you would simply use an app or a website to order the pizza i'm jealous of that um but we will see that the patterns are kind of the same okay The one important thing here is also the feedback loop is not the result you want to have. So just that they say you will get a pizza doesn't mean you stop being hungry, right? You haven't yet have a pizza. The pizza is the result. And the task of pizza making, it's what I call long running, right? Because they have to put it in the oven. They have to wait for whatever. five minutes, 10 minutes, 15 minutes depends on the pizza place and the utilization and so on and so forth. And then you get the pizza. So there is an element of what I call long running in here. And that means I always have a delay getting the real response. And we will see also later on that this is a quite typical pattern now that we see we might have a first step that is in a way with a feedback loop. maybe synchronous, maybe not, but in a short time frame. And then we have a long running step to get the real result. And you can easily imagine that. And you can also imagine the other way around, like saying, OK, what if I use, for example, synchronous blocking behavior for the result? And the way I always imagine that is walking to a local bakery around the corner here, which is not that picture, but I think, you know, these kind of like small little. um bakery shops and they normally also have a coffee machine in there and if you order coffee there that's so good as blocking behavior right you you order the coffee and the uh the the person behind the corner uh counter turns around walks through the coffee machine press the button needs to whatever clean up that the mess first and um the other person doesn't do anything in that time You're not doing anything. You're blocked. The queue behind you gets longer. If you order 10 coffees, that's a fun exercise to do and so on and so forth. So it's pretty easy to imagine this is a bad user experience, actually. And it also doesn't scale very well. So we don't want to block for the result. And if you imagine coffee, if you're not so much into pizza at that time of the day, if you imagine coffee, you could imagine something like Starbucks. quite good article blog post actually from Gregor Hoppe. You might know Gregor Hoppe from the Enterprise Integration Patterns book. It's relatively old, but I still find it a really good resource. He wrote about why Starbucks doesn't use two-phase commit. That's the title of the article, but it really shows exactly that. So you have that first step of ordering of paying, which is kind of in a way synchronous blocking. right and then the order goes into a queue um it's it's written on a piece of paper basically and then the baristas doing the coffee they're scaled independently of the cashiers um and then at some point in time the coffee making is also a bit long running um you get the coffee and they call out your name um that's kind of a correlation identifier so they make sure you get the coffee. And you can already see that the scale is much better. It's a better user experience. And what I also find very interesting is you have more options to adjust the user experience or the customer journey, if you will. And a lot of companies now do that. You could replace the step of paying and ordering with a simple app, which you do already when you walk in the store or probably even before. adjust the journey because you now have flexibility to do so. And it would be really hard to automate, for example, the coffee making. But it's relatively simple to automate the order. OK. So synchronous blocking, non-blocking, we have the feedback, not the result. Result is later. And we have long running. And long running is the topic for my talk today. And if I say long running, and I like the term, actually. And so I keep using that very often. But if I say long running, it normally means that I have to wait for something like the pizza in the oven, right? It's not that I actively do something like massage the pizza so it gets ready. It's like I just wait for the time to pass or same with the coffee. I might need to wait for the coffee machine to warm up, for the coffee to get through and whatever. So there is normally a lot of waiting involved and that's... what i want to talk about and that's what i mean when i say long running it basically means waiting and there there are lots of reasons why you want to wait um some typical business reasons we will come to technical reasons later but some typical business reasons so requirements are there's human work so you might want to push certain tasks to users in order to do something and then you have to wait for them to get work, to pick it up, to do the actual stuff and so on and so forth. There are a lot of moments where you have to wait for responses like whatever you want to open up or you want to apply for a new insurance policy. You do all the paperwork and then you have or you get all the paperwork from the customer and then you have to ask for some additional papers or something to sign off and then you send it to the customer, need to wait. for the response. And that will not happen in milliseconds. It will be much, much later. Or you simply have to let some time pass. That's the, I put the pizza in the oven and then I simply wait 10 minutes. Or one of my personal favorites was a startup actually that automated a service which was 100% automated. So that ran through in seconds, but they added a, they wanted to have a wait state in there. for I don't recall exactly 30 minutes, 45 minutes, because they sold it in a way that it's kind of crafted, that a human does all the work. They wanted to have that impression with the customer. Oh, somebody looked at that and did a lot of work. So it should take longer. Or bias remorse was an example I liked also very much that there's kind of scientific research that shows that after you bought some something you might regret it and then you might cancel your order right away. So within the first minutes or within the first hours there are the most cancellations actually happening and if you start fulfilling the order right away the cancellation might be really hard because you probably already picked stuff from the warehouse for example and then it's really hard to cancel the order or the parcel might already be on the way and then you have to return the parcel. so on so forth it's much more expensive so it might make sense to wait an hour before you start to fulfill the order and so on so forth there are hundreds of of good examples when you have to let some time pass and implementing that is actually painful in java and all the other languages i picked java here because we're on the java day of course um why because you need to have state right you have to remember that you're waiting it's kind of a I'm not sure if you're even thinking of having a thread open for days to wait for something. Of course not, right? So we have to write that state somewhere in a database, for example. We have to keep it persistent so we remember we're waiting. Now, you could argue, and I see that happening a lot, actually, in a lot of scenarios. OK, that's easy. Let's add a database here. Let's put an order in the database. Let's give it a status. column like order entered or the paid or the fulfilled. And that's it, right? Then we have our state. Where's the problem? Then we can wait. And the thing is, as soon as you start doing that, you get subsequent requirements around the state handling. Like you want to see what's currently going on. Where are you waiting? Where are you waiting for too long? probably not only want to see that, you also want to do some escalations automatically, for example. Oh, if the customer doesn't pay within two days, we need to remind the customer. Or after five days, we cancel the order automatically. There are a lot of these examples. And if these escalations, for example, if they're failures during escalations, you need, again, some possibility to do something with that exception you might get. because there's no client available anymore. You cannot just re-throw an exception. You have to have a monitoring capability. If you have long-running things like think order fulfillment, you always have orders circulating in your system. If you want to change the logic how to fulfill orders, you have running orders and you have new orders with a different logic. So you need kind of a versioning capability. And it's very often domain logic we're talking about that. I'm getting to that later in the talk as well. And you have to do that reliably and so on and so forth. So it's actually not that easy. And in my career, I'm doing that for 15, almost 20 years now. I saw so many homegrown kind of workflow engines. I get to that in a second because people underestimated these kind of requirements they get around the state handling. So it's all about how can we solve the problem of long running without adding accidental complexity like the database table let's get you like this and that and that um that must be easier and it is easier the one thing i want to want to say here full disclosure i'm like every human being i have an opinion i might be biased um just making it transparent so i'm co-founder of camunda been has said that very briefly in the beginning um we're an open source a process automation, process orchestration vendor. We're out there for quite a while. But that's also the perspective I'm thinking about problems. So I'm kind of, since the beginning of my career, I'm doing something with workflow engines, orchestration engines, and these kind of things. So that's my bias. I'm thinking through the lens of workflow engines about problems. The most important thing about that slide is if there are any questions afterwards, any feedback, there's my email address. You find me on LinkedIn. Just connect and ask whatever you want to ask or discuss whatever you want to discuss. Good. So keep that in mind when I keep talking. So obviously there are solutions for long running. They're called workflow engines. It's pretty still the most popular term, I would say, but you might also call it process engine or Process orchestration engine is kind of the term of the day. But it's normally all means the same. It means that you have a piece of technology that can express certain processes like steps in models and very often graphically. Version them to solve the version problem. Keep them durable, persistent, have all the tools around that for management operations and so on and so forth. And also have scheduling capabilities, right? So that's the idea of a virtual engine. Okay. In order to make that more transparent or more lively, let's quickly look at that. And I'm going into a quick example. And you have all the code on GitHub. You have the link here. Keep it open for a second so you can still scan the QR code. But I'm happy to also put it in the chat later. And if we're going there, right, I'm going to my IntelliJ in this case, I have a Spring Boot project here. So nothing very surprising, Maven and Spring Boot and Java. And what I have as part of that process is not much. I'm depending on something called CV. That's our workflow engine. I'm basing this here on our SaaS offering. So I have created a cluster in our SaaS offering. That's one way of doing it. There are others. I'm not going into details of that. Having that running in the background, having a client SDK connecting to it, having a couple of properties, how to talk with my SaaS instance, and that's kind of the setup. And then I have a process model as part of my product here, which is customer onboarding. That's the example I use for the quick tour. BPMN model, business process model annotation, it's an ISO standard to define these processes. And then I can basically define the different steps I want to do in order to onboard a new customer, which we have in almost any industry. It might be opening a bank account, applying for insurance policy, getting your mobile phone contract, whatever, register a new domain, whatever you want to pick as an example. They always have onboarding processes and they're almost always kind of the same. You score or you Do clearing. You approve or disapprove. If you approve, you provision the thing, and then you're done. So I want to do a very easy scoring, approve the customer. That's a so-called manual task. That's a service task, a system task. And that's more or less it. I need to have a look on the time. So I'm not going into all the details here. I want to highlight one or two things. So one is... um scoring the customer what does that mean how do i score the customer um the one of the key key ideas here is i glue the the graphical model in the workflow engine to actual code so score customer has a so-called task type and that is his connection to some java code i have here so it's a spring bean it's a so-called job worker The job worker makes a connection to the workflow engine. And then that is the task type. I could also refer to it here. But the default is just using the method name. So that's the connection. So now when I start this application, it knows that whenever I have new workflow or process arriving in that state, this code should be executed. And if that's done, the process moves on. And then I go into a user task, which um then has a form connected to show it on a kind of task list okay cool um last thing to quickly show you and then i also have a rest api here in front of the whole solution which can start new onboarding processes where you say okay um i basically put some data probably copying that from the request and start a new process instance and then we're good okay um With all of that, I also added, and I'm always happy to show that, a very simple HTML page. That's my level of HTML I'm happy to do, with a bit of jQuery to just kick off that REST API. So it's not because I proposed this as the best web UI you can ever do, but I like that it shows the simplicity of the end. what you what you actually need and then we end up with a beautiful ui which like that um where you for example can open my new bank account and probably enter some data and then i i push the button in the background and i think stream yard or here okay i think we're back um so it's it started a new process instance successfully. And then you have all these kind of tools where you can look into, hey, what's going on? OK, I have a new process instance currently going on. It waits for approval. It already did the scoring. There's some data attached and so on and so forth. You have a task list that's exactly about all the long-running things. So I see I need to do an approval. I see the data. I can pick it to do it personally. And I can confirm this should be automatically processed. And then this kind of moves on. I see that and operate in a second and we see how to take it through. Okay, good. Yeah, right. And sends a welcome email. I haven't showed that just a very quick thing, but there are also, if you don't want to connect your own code, you can also use pre-built connectors. And I've built my customized one for a different demo actually, which always sends a SendGrid email. So that should... probably also have happened, or it doesn't, probably my SendGrid. Yeah, okay. I haven't added the SendGrid secret in the preparation here. But that's not a problem. Actually, it gives us a good opportunity to see what happens if something fails. We talk about that in a second. You see... um there's an incident in my process you see a big red light you can dive into that what's what processes have those incidents you can see where it's at you can see that's what i had early on you can see the full message you can retry it so i could add the secret now and then i could retry it and it just moves on okay and just for the long running part i could also add now um things around the notion of time so i could say if approval takes too long time a boundary event um non-interrupting the dashed line says okay if that takes too long i do something else additionally so i keep waiting for the approval but i um whatever oh i can't say kick ass it's not politically correct but remind somebody to do the approval. I could probably do a user task again, assign that to a manager or whatever the role is. I save that. I can restart my application. In this case, it's built in a way that it does an auto deployment of my process model when the Spring Boot app comes up. It deploys it. So I should have a new version of my process. I'm seeing that in a second. And then if I submit a new application here, it's interesting. My super fancy HTML page seems to bring my Chrome down when I have StreamYard open. I have to look into that sooner or later. What's happening here? Oh, we have an exception here. OK, my process model was not correct. I didn't configure the timer. So if I say a timer, I have to say how long. So we could say a duration, period time, 10 seconds. Okay, made the product model correctly. Start again. I'm not going into any product details. We have actually a linting tab in the modeler that would show me a lot of those things as well. We should be there now. Okay. Refresh. Yes, I can start a new application. And then if I go back to operate, I'm looking at my onboarding process. I have the new version. What you can see is that the old process is still running in the old version, for example, and the new one is here. And if I talk for another couple of seconds, I see that this is also escalated. And yeah, this kind of thing. I hope you get an idea of what I mean by workflow engine, what I mean by workflow or orchestration technology to solve long-running problems. OK. Yeah, that should be OK. I just wanted to give you an idea so you can better connect that with what I'm talking about next in your mind. Just to recap, as the solution architecture, what I had was just a simple JavaScript boot app. I had an SDK. I used Commodore 8 as a SAS version. And then I had custom code to connect to an REST endpoint, for example. I don't have that on the slide, my connector to connect to Zenquit, for example. OK, good. So we talked about these business reasons why we want to wait, which are very often front of mind when we say long running. But there are also technical reasons, which very often are also motivation to go into that direction. And for example, what I said in the beginning, if we do asynchronous communication, for example, with messaging, and then we might wait for responses. Like we want to see that the feedback email from the pizza order arrives in a certain time frame. And if we assume that's not milliseconds, but probably minutes, then this starts to get a long running problem where we don't want to wait in a threat anymore. And this gets especially interesting if we look at failure cases, if the response doesn't arrive, because then we very often have to do something around that. And very often it's long running. I'll give you an example in a minute. Also waiting for availability that might be in the synchronous case that we can't call the other service because it's not running at the moment. What do we do? We have to wait probably for it to become available. That's kind of the situation. We don't get a response because the peer might not be available at the moment. OK, so that's a very, very common thing. I already showed you that in a way with the with the incident. the peer is not available you get an incident in the workflow engine you have automatic retries and so on so forth in the matter of time i'm not going too much into that you i'm happy if you ask questions around that then we can might use the q day to have a deeper dive into that i wanted to um again go on the metaphor level a little bit to to also give you an idea why you probably want to use something like that. And one of my favorite examples is this. In a live setting, I normally ask you who has used an airplane before and then most of the people raise hands, which is good because then you know the procedure, right? You normally book a flight and then you get a check-in invitation a day before your flight takes off, right? And you should check in. And I got that for a flight to London actually. quite a bite back, pre-pandemic actually, this example. And when I did that and I wanted to check in, I did everything, I pushed the button and then I got this error message basically saying, hey, there was an error while sending you your boarding pass. So I couldn't proceed. What's the normal reaction? I can't ask you, unfortunately, but it's also for computer scientists, very natural. I retry it, right? I tried again. My wife always gets a bit crazy. Why should it work if I try it again? Trust me, try it again. But it didn't work in that case. What's the next thing you do? You wait a couple of minutes and try it again. And it didn't work. What was the next thing you do? At least what I did was, because it was still 24 hours till the flight, so it was not in a hurry. So I made an Outlook entry in my calendar to remind me to retry. the check-in in, I don't recall exactly, four or five hours. That's exactly what I wanted to get to. I need that retry, but in a long-running, long-running manner. And the situation I envision, we're not working with Eurowings, but we're working with comparable companies. And the situation I envision is that we have a couple of different services. um at play microservices or whatever but we have different components we need in order to for the check into work so for example the 3d barcode generation or the output management to generate a pdf or whatever or um i think it was british airways that still offers to send you the boarding pass as fax message so then you have that component somewhere and we see that all the time right we need more and more components in order for one functionality to be delivered and what i think happened is that some of those simply broke and that's in a way fine i mean the the situation we are approaching nowadays with more and more services more and more saas services more and more components we need in order to to deliver functionality is very different to the monolith we we had built 20 years ago where we could say okay, we put a lot of hardware in it, we just make it highly available. That's not the mindset we have anymore. We say we have more and more components and some of them will always be broken, right? Or the network towards them will be broken. So something will happen. And that's okay. We need to deal with that. What we have to make sure is that it doesn't like really brings down like the whole system completely. And we need to be resilient. with those failures and thinking about that in the in the check-in case is if there is a problem with the barcode generation what we actually um should do is we should deal with that in a very local scope we should not as the example was we should not bring that really up to the user experience where you say oh we can't check you in there's a problem any kind of problem but you you have to deal with it That's a bad design. That's a bad architecture. And in this case, it's very obvious because it's me as the customer in front of the screen. But this is kind of the mindset I'm still seeing with a lot of, let's say, service to service interaction within organizations. The mindset should be very different. These should really be all right. And that's a funny part of that story. it's really the same trip when i had my onwards flight from london um i had the same problem with easy chat um we have technical difficulties right so they couldn't check me in but the interesting part is um and that's different they give me the work instruction to work with that so please log again hey retry if that doesn't work please try again in five minutes okay increase the interval and And I love that most. We do actively monitor our site and we'll be working to resolve the issue. There's no need to call. It's your problem. Please sort it out. It's not our thing. And I mean, that's attitude-wise, I think, the wrong thing. They should be much more like, OK, we have technical difficulties. We're sorry. That can happen. We try everything we do. And then we will let you know. right we don't make it your problem it's our technical glitch we make it our problem we solve it so like looking at the architect that would mean the check-in service should solve that on itself right and then it never leaves that scope it makes a cleaner architecture because the web ui or what whatever other component is never really influenced by um these like technical problems um check-in has. But that also makes the check-in service at least what I call potentially long-running service. So it might take time in failure cases. It might deliver a response in a millisecond if everything is great, but it might not. And it's a much better architecture to deal with that, to think about that possibility of long-running. And very often when I say this or when I discuss that, I get the um if you the headwind if you will um yeah but the customer needs a synchronous response and i discussed that with especially one customer i i recall um very very thoroughly they were in that no we need we need a pdf people want to print it out right away we we need that and if you think about that it's i i find that 100 nonsense because if the choice is you either get a synchronous failure message where you left alone with, or you get a proper result later. I mean, I know what I would take, right, as a user, because there's, yeah, I come back to that later on as well. I mean, you can still provide the PDF to print out if everything is fine. Okay. I don't have any problems with that, but it should be treated as a special case. I mean, even if it, it's hopefully the normal case and you should at least think about that this is not the 100 case there are a lot of edge cases where this will not happen and it would be good to still operate properly quick detour which i find important so we talked about retrying things if you implement retry also implement idepotency and that's one of the big obstacles or challenges i see companies having when we start adding long-running capabilities, adding retries, they very often deal with services that are not idempotent. Let me quickly explain idempotent. So let's assume you have a payment service that wants to charge a credit card using a credit card service. Then you have an API call to charge. And depending on basically how the API is designed, that could mean that you charge the credit card just giving a credit card number. and the amount to charge. This is not idempotent because if you retry that, the target service don't know if it gets the same call twice, if it's the same call, just retry it, or if you want to do just the exact same charge twice. It might be right, especially if you look at whatever. consuming music or videos on demand, and you might buy two pieces of music for the same price or whatever. The thing is, if you have an API design like that, it's always guesswork. It's best effort to see if it's a duplicate or not. And that makes it sometimes incredibly hard for the service to detect duplicate calls. And that makes it hard to decide that you easily go for retry. So you should always, always, always, whenever you have the choice, doing APIs that have some kind of, for example, a transaction ID where it can easily detect duplicates, where it's idempotent because you can call it again, and the credit card service can immediately know if it already charged for that transaction ID or not. And the trick is normally to create that kind of artificial ID very, very early on, so probably already in the UI. your UID and just pass it along. And then everybody along this chain knows if it's a duplicate or not. And that makes it incredibly much easier to apply retrying on a broad scale. So that's in a way not long running, I agree, but I find it an important ingredient to move along. I find this an interesting example. I discussed that with one customer quite a while back in depth, actually. So assume the situation where you want to charge the credit card, right? And assume you're using a workflow engine for that, because then you can do retrying. And you can do retrying in a stateful manner, saying, like, I retry that for 10 minutes or an hour or two hours. So that it's the same like the check-in component retries the barcode generation unless it's successful. uh until it's successful um right so you can you could do that hopefully the credit card service is um but still um at some point in time you might give up right hey we we tried it for an hour no way um the payment failed and if you do that there's one interesting thing also to keep in mind um if you have distributed system network communication it's impossible to differentiate certain failure scenarios. Like if there's a network error, you might never reach the service provider. You might have reached it, but it exploded during doing your request, or it might have committed its transaction, but the response got lost in the network. And you have no idea which of those happened. There's no way of doing that. So, and it's independent if using REST or messaging or Kafka or whatever. There's no way of knowing that. And that means in that scenario, you retry that five times, you never get a response, and you say the payment has failed. You might have actually charged the credit card. Technically possible. If you don't have an IDA ported service, you might have charged it five times. So very often, it's interesting to look at those cases as well and say, OK, we want to probably refund or cancel the charge. So there might be things we need to do in order to clean up. It might not be the best way to do that with the workflow engine. I'm not saying that. There might be reconciliation shops running at the end of the month or whatever. But it's one interesting possibility at least to have in your tool band. And then because the service was not available when charging, it might not be available when you want to cancel the charge. So again, that might be long running. And that adds a lot of reliability to what you do it adds a lot of resilience to to what you do and then the example i had early on then you can still build a service which gives you hey payment received as a synchronous blocking response via rest if everything works fine but if not you probably just get a rest code 202 which means accepted we got your request we sort it out and you need a callback mechanism or messaging mechanism to send the response later on. Very, very important pattern. We don't have time to look into that today, but there's also code showing all of that in more depth available on GitHub. Cool. So for me, one of my key points I wanted to get to is like long running capabilities or the possibility to implement long running easily are essential to design good service problems. And this is from my perspective, very often forgotten. And I make another quick example. I'm an example person. I'm sorry. I hate theory and slides. I always look into examples. Let's assume you have a booking site for whatever, train ticket, flights. You have a payment service and you have the credit card service. And the booking says, hey, retrieve payment for me. Might be a rest call, might be messaging. It doesn't matter for the sake of the example. Payment says, okay, in this case, I wanna charge the credit card. So I charge the credit card and then I get a rejection method. The credit card was rejected. Now, there are a couple of different possibilities what the payment service could do now. It could either forward that rejection, credit card rejection to the booking service. Probably not the best idea, but it could do that. Let's simply assume you have kind of an architecture like that. And then you get a business requirement of saying, OK, the credit card was rejected. The customer can provide new details. And this is very often if you have your payment details in your account and probably the credit card expired and you want to give people the opportunity to update the credit card and probably with a good user experience. A common example is something where you have a subscription like GitHub. If GitHub renews, your credit card is expired. They send you exactly an email saying, OK, credit card expired. We want a chart. Resolve that within, I think, two weeks is the time frame they give you. And then everything is fine. I would love to see that also for other things, like I book a train ticket. And the current user experience is if the credit card is rejected, my whole booking is kind of gone. It just says, oh, payment failed, and then I have to redo everything. It would be great if they say, OK, we got your order. It's there. We reserved your seats. They're blocked for another hour. But your payment failed. If you update your payment method and it goes through in time, we're all good. We send you an email. If not, we can still cancel that pending order you have. That would be a very different way of designing the customer journey, but a much better user experience. One of the key things in order to make that happen would be really here to be able to implement those requirements properly. For example, what I see happening very often with this requirement, it ends up in a booking service because the booking service is long running anyway. They have stayed, they have solved that. So, hey, let's put that requirement there. And then the booking service grows and grows and grows. And it does things it's not. responsible for because payment is responsible for payment. I think that's kind of obvious, right? So we should actually write and then you end up with things like this is Sam Newman. He wrote microservices books, pretty famous ones. And he says along the lines of criticizing orchestration for weird reasons to some extent. I agree with him to disagree on certain things. But he says, for example, then you end up with a few smart God services tell and they make CRUD services what to do. And for me, that's very often the reason because only those, let's say, bigger services have the possibility to implement long running capabilities. Then you end up with this. But it's not a given. So it's much even much better to say payment should solve that because then payment could be potentially long running. right they could sort it out and they never ever give the ticket service or the auto fulfillment service some response saying credit card expired you only get a response that payment was either received successfully or ultimately failed so you don't even retry anymore and that's a much better design it gives you much cleaner services And then you might end up with a long-running service also in payment and probably even in the credit card processing. So that's kind of where I want to get to. So if you have the possibility of or the capability to implement long-running things easily in your architecture, you can use that at the place where you need to and then you can build a much cleaner architecture overall. Also looking at microservices, for example. Because then you could, for example, extend payment options and so on and so forth. We're not looking into that too much today. So, right. I said all of that already. And the only thing I want to add here, which I find important because it's asked so often, and it doesn't mean just because I use a workflow engine or an orchestration engine that I get a monolithic process or whatever. There's very often misconception about that. And you can have processes in all of those services distributed and so on and so forth. OK, if you want to know more about that, let me know. And very last but not least, graphical models, really, that's the other conception I sometimes get. We want to solve these long-running problems. We want to code. We are Java developers. Why should I use graphical models? And the thing is, I have really, really, really good experiences with BPMN on all levels with different stakeholders. So first of all, it's living documentation, the process model you have, the graphical one, it's running code. So you always have that place where you can understand the thing. And especially with long running things, you have to talk through them with more and more stakeholders, actually, because there are questions of why do we wait there? How often do we cancel there? What's the right waiting time? What can we do if that's not happening? And so on and so forth. We have typically very often discussions around that on a business level. And this is so valuable to have these kind of graphics you can look into, which are not wishful thinking, which are not documentation. They're really code. They're used at all places. That's a screenshot from when we run test cases, you can create these kind of reports for this. for a specific test run. So you see what's happening there, what's the test coverage or all the tools you saw early on, what's happening. We have a lot of like also statistics and you can see bottlenecks and whatever. So there are a lot of insights you can draw from that, which makes it very, very, very powerful actually. And I have probably, I personally never understood the reluctance to use graphical models, but I I just want to invite you to give it a severe try because it's simply very powerful. And a lot of last thing, I promise, that's the thing I see at the moment being very, very important. We do a lot of changes on a technical level, microservices, cloud SaaS services, and probably even serverless things or whatever. So we change the way we're building and deploying. But it influences the user experience and the customer journey. It also influences the possibilities I have doing that. I had a couple of examples early on. And that means a lot of those things needs to be discussed with the business people because it's about requirements. What's the consistency level we want to guarantee? How long can we be inconsistent in certain cases? That's a business decision. And that's why it's another reason why I think having graphics that help us communicate that are essential. So let me recap. So we need long running. We need technical capabilities to solve long running, right? Process orchestration engines are a great fit, obviously. But there are others as well. Go get a look yourself. And that allows you to design better service boundaries and embrace asynchronicity. It's not a problem anymore. It's actually easy to handle. And it really results in a better customer experience at the very end. It's also fun to use. If you want to learn more, you can either get to our homepage and try it out. You can get to the GitHub examples I added to try it out. You can, of course, read my book. And that's all I have. Thank you so much. Let's see if we can be in a bag. Thank you so much, Bernd. And hello again after this wonderful talk. I could relate to the comparison of the pizza in the beginning. I had yesterday a pretty good one. Happy to you. All right, Bernd. We still got some time left and some questions. So I would say let's jump into a quick Q&A. Let's go. The first two questions are, I think, easy and are kind of connected to each other because first one would be, what is the name of the product and as a follow-up question is it open source can i use it in production for free yeah so the name of the product is like the name of the company so it's camonda so that should be easy i had on the slides here a m And yes, it's open source available at the correct term because there are some classes in there. There's a whole discussion where the difference is if you're interested in that, shoot me an email. But it's available, not all parts of it. So there are parts of that which are not free for production use, which is the typical upsell opportunity you want to have as a vendor. But there is a path to use it completely free in production as well. Yeah, that's a quick, quick answer to that. All right. Thank you, Ben. Next question. What are the cons or the trade of companies need to be aware of when implementing these solutions at scale? That's a broad question. Let me I mean, as it is a very broad and high level question, I can also only answer a high level in a way. And the trade off is always. You basically introduce a new technology that always comes with a bit of overhead, a learning curve. You need to evaluate the technology. You probably need to find your way of using that in the company. That's always a bit of an overhead. That's kind of the cost you have at the same time. You have a gain, which is typically around saving effort, because very often people go into solving that on their own. They're writing code around that, which goes into technical. accidental complexity, which simply consumes time and resources as well, especially on the long run maintenance. So that's the obvious one. The other not so obvious one is that very often this introducing such a technology also has kind of a transformational power, let's say, to better understand processes, to better think about long running things, to also better align that with the business stakeholders. So that very often helps actually a lot of our customers to also improve their processes quite a big time. And that's harder to put in correct numbers or concrete numbers. And the scale one, just to add on the scale one, it's probably not part of that exact question, but the scale is something we... we really focused on a lot over the last year. So first of all, with the technology we have now, we have a horizontally scalable tech. So we can basically go into all the use cases. So we have customers running really concrete money transfer like big banks on Camunda. So we can get into the multiple thousand instances per second easily so that that works. So that technical scale is solved by now. And we're also currently have customers that run hundreds of different processes on Kamunda. So we are also working with them to get all the best practices around how can we help organizations to get to that scale? Do I need some whatever internal expert, internal center of excellence? How can we set up those projects? What are good, whatever golden path or templates we want to use? So also on the organizational scale side, there's a lot of material and maturity at the moment. Sounds reasonable to me. Thanks for this answer, Bernd. And let's move to the next question. How accessible and user friendly are these solutions? Are there further resources or best practices? A lot. Yeah, of course. It's probably more the problem of too much stuff, not too little. No, I mean, again, very generic question. So it depends on what you mean. with the user. So user, if I look at Camunda as a product, the user could be the developer using it to build a solution. Then it's more about the developer experience where I would say relatively simple to use. We still have some homework we are currently on. So I hope we're even getting much better over the next six months, but it's still. really easy to get something going, especially if you use the SaaS service, there's a trial version, so you don't have to set up anything locally. If you look at the end user working with the solution later on, so if you build an onboarding process, there might be people in the company needing to do whatever approvals. It's a very different story, and that's where it really depends on what you're building. So a lot of customers are building their own UIs or they're embedding that in existing um ui's or you could use slack as an end user interface if you like to but they are of course also the the tools you saw earlier on like our own task list um where um where people can work with and um so it's hard to comment on that on generic basic what's the user friendliness of that i see thanks for this answer as well band next question does camunda store its state in a database or are multiple databases supported? So we moved from Camunda 7 to Camunda 8, so a new major version two or three years back. And the Camunda 7 version was based on a relational database, and there we supported kind of the typical ones, actually. The thing is, with relational databases, we hit a limit in scalability, also in resilience. So for example, if you want to, we have customers who need I'm not sure if they want, but they need a geo redundancy. So they have data centers, for example, in the US and in Europe, and they replicate across them. And doing that with a relational database and the architecture you build upon relational databases was actually limiting us in scale. And also like running 10,000 of process instances per second is also pretty, pretty hard with a relational database. So what I want to say is we switched. to a different architecture it's kind of if you want if you will event sourcing underneath where we do our own state um it's binary on disk and then we replicate ourselves to our peers um and there you don't need um external dependencies for all the and then right and this is for scalability and resilience and we now can do um even those geo-redundant scenarios so that adds latency but it doesn't bring throughput down so that's actually pretty impressive the one dependency we still have for all the tools where you can look into things like this operating tool or task list there we're using elastic search and then thank you band and as a last question um since it's kind of a hot topic right now have you already worked with java 21 if so what do you think about it Are there features missing, in your opinion? I'm probably the wrong person commenting on that. Honestly, I'm normally the one fighting the fight while we still want to want to support Java 8 for certain client SDKs because we have big customers that are really stuck on old versions and very often. But I'm looking at it from the Viren vendor. and there you have the differentiating of providing building the core product there we're going with the latest java versions but we're also providing sdks to our customers and those are normally reluctant to go with the latest and greatest because it doesn't give us a big advantage and we want to be compatible so i'm i'm normally very often in the role of the old is still good so i'm i'm not the best to comment on that question all right next time we will not talk about yes yeah i prepare myself next story for that all right and with that we also come to an end for this q a thank you so much for the very informative talk and for the q a was nice having you and hopefully until next time yeah yeah thanks for having me bina and everybody else have a great time today

Surviving modern architecture requires you to navigate challenges posed by long-running processes. Using more distributed systems intensifies complexities and poses more issues with remote communication. Furthermore, effective domain boundary design requires addressing business requirements around waiting. I'll illustrate these challenges through real-life examples, emphasizing their importance. Ignoring these issues is not an option! Exploring solutions, I delve into the role of process orchestration. A new generation of microservice orchestrators and workflow engines are designed for infinite scalability, which offers capabilities to handle long-running processes efficiently in any situation. This empowers software engineers to design better domain boundaries, resulting in better architecture. You will see real-life examples and sample code (available on GitHub to play around with yourself) to help you decide and adopt long-running processes in your architecture.

Welcome to WeAreDevelopers, Europe’s #1 developer community.
Subscribe to our channel for in-depth tech talks and conversations with developers about their journey and how they built their career.

Create a free developer profile and access skill-matched job opportunities:
https://www.wearedevelopers.com/en/developer/sign-up