Are You Done Yet? Mastering Long-Running Processes in Modern Architectures

and good morning from me right I talked about long-running not so much about exercise actually we want to start talking about food first probably more enjoyable if you if you want to order pizza there are a couple of ways of ordering pizza you probably have ordered a pizza in the past if you live in a very small city like I do if you ordered pizza what you do is actually you call the pizza place that's a phone call and that it's very i started with the very basics apologies for that but we need that later on okay if i do a phone call that's synchronous blocking communication right because i'm blocked i pick up the phone i have to wait for the other person to answer it i'm blocked until i got my message and whatever and whatnot but if then the person answers me i get a direct feedback loop though Normally that person either tells me they make my pizza or they don't right they can reject it But they I get a direct feedback But I'm also temporarily coupled to the availability of the other side So if the person is currently not available to pick up the phone if they're already talking on another line They might not be able to take my call right and then it's on me to fix that So I have to probably call them again in five minutes or really stay on the line to do that, right? temporary synchronous blocking communication what would be an alternative i mean i know you could probably use the app but again i can't do that where i left sorry for that you could send an email an email puts basically a queue in between it's asynchronous non-blocking communication and there's no temporal couple guys can send the email even if the peer is not available even if they take other orders how does it make you feel if you send an email to your pizza place Right, exactly that. Because there is no feedback loop at all. Do they read my email? I pick up the phone to call them if I said no, whatever. But it could be. It's not a technical restriction that there is no feedback loop. They could simply answer the email saying, hey, we got your order and you get your pizza within whatever, 30 minutes. So you can do a feedback loop again asynchronously. It's not really the focus of today, but I have another talk also. talking I think 40 minutes about the that this is not the same you can have those interaction patterns decoupled basically from the technology you're using doing that but so gonna split in communication is on the next slide even if I do that independent of email or phone it's important to distinguish that the feedback loop is not the result right I'm still hungry They told me they send a pizza. I'm probably even more hungry than before. The result of the pizza. But the task of pizza making is long running. So it probably goes into a queue for being baked. It goes into the oven. They hopefully take the right time to do that. They can't do that in a rush. Then the pizza is ready and it needs to be delivered to me. So it's always long running. It takes time. It's inherently there. And that's actually a pattern we see in a lot of interactions, not only for pizza, but for a lot of other things. And I go over that a couple of times in the talk. So we have a first step that's synchronous blocking, but we have an asynchronous result later on. Could you do synchronous blocking behavior for the result in that case? Probably not such a good idea. If you take the example not of pizza, but of coffee, which probably fits the timing better here. If you go to a small bakery, that's a random picture from the internet about a German small bakery. If you go there and order coffee, what happens is that the person behind the counter takes your offer, takes your money, basically turns around, going for the coffee machine, presses a button, waits for the coffee to go out of that, going back to you, gives you the coffee. So it's synchronous blocking, right? They can't do anything else. I can't do anything else. We're waiting for the coffee to get ready. If you have a queue behind you and if you're in a good mood to make friends, you probably order 10 coffees. It takes a while. So it's not a good idea. It's not a good user experience here and it doesn't scale very well. Right and the coffee making is relatively quick compared to the pizza making other things. Okay, it doesn't have to be that way. There's a great article from Kiko Hooper. I think Kiko is talking tomorrow also probably not on that but he's here as well He called it and star Starbucks doesn't use two-phase commit. So he talked about scalable coffee making at Starbucks where you also like separate the two things the first step is a synchronous blocking thing i go to the counter order pay and then they basically ask for my names or a correlation identifier and then they put me in a queue saying okay to the baristas make that coffee for burnt and then the baristas are scaled independently there might be more than one for example doing the coffee and then i get the coffee later on and that scales much better It also, and that's another thing you can recognize here, it also makes it easier to change the experience of the whole process. So a lot of the fast food chains have started to replace, not fully replace, but replace some of the counters or the humans behind the counter with simply ordering by the app. Because that's very easy for the first step, but not so easy for the coffee making. i mean okay there's robotics also for that there are videos in the internet how you can do that but it's not on a big scale so normally the baristas are still there the coffee making is still okay so we want to like distribute those two steps and if i with that in mind if i come back to long running when i say long long running i don't refer to any whatever ai algorithm that runs for ages until i get a result no i'm basically simply uh refer to waiting Long running for me, it's waiting because I have to wait for certain things. That could be human work, the human in the loop, like we just heard, because somebody has to prove something, somebody has to decide something, that are typically things, or somebody has to do something. I'm waiting for a response. I send whatever inquiry to the customer, and they have to give me certain data. They have to make their decision. They have to sign a document, whatever it is. So I have to wait for that. And both of those things, by the way, are not within seconds. They can be within hours, days, or even weeks, sometimes even longer. Or I simply want to let some time pass. The pizza baking is one example, but I had a lot of other examples in the past. So one of my favorites was a startup. They did kind of a, let's say, manufactured service, which was completely automated, but they wanted to make the impression to the customer that it's like a human does it. So they waited for whatever a random time between 10 and 15 minutes for example to process a response But they're also more serious examples like I could bring you a whole list So why is waiting a pain? It basically boils down to because we have to remember that we are waiting It's important to not forget about waiting and that involves persistent state Because if I have to wait not only for seconds, but minutes hours days or weeks or a month I have to persist it somewhere to still remember it when somebody comes back. Persistent state, is that a problem? We have databases? We do? Yeah, right, but there are a lot of subsequent requirements if you look at that. So for example, you have to have an understanding what you're waiting for. You probably have to escalate if you're waiting for too long. You have versioning problems like if I have a process that runs for, let's say, a month and I start that every day a couple of times, I always have processes in flux. If I want to change the process, I have to think about already running ones and probably do something different for them than for newer ones, for example. I have to run that at scale. I want to see where I'm at and a lot of those things. And the big question is how do I do that? How do I solve those technical challenges without adding accidental complexity? That's what I'm seeing actually quite often. I wrote a blog post, I think more than 10 years ago, where I said I don't want to see any more homegrown workflow engines because people stumble into that. Like, oh, we simply have to write a status flag in the database. then we wait that's fine and then they start oh we have to remember that we have to have a scheduler we have to have an escalation oh people want to see that so they start adding stuff and that's not a good good idea to do um probably one word of warning just to make it transparent everybody's opinionator of course i am as well so that's my background i'm working on workflow engines cross engines orchestration engines however you want to call them for like always save almost all my life at least my professional life um i co-founded kamuna we're in a plus orchestration company and wrote a lot of things in the past about it i worked on a couple of different open source workflow engines as well in the past the most important thing here is there's my email address there's my uh okay the twitter handle is that we can confess that but you find me on linkedin if there's anything you want to discuss um hunt me down later on I'm also here today, of course, so you can ask me anything. One of the components that can solve these long-running issues is you might want to call it workflow engine. As I said, the naming is not always very precise. You're currently more going towards naming it an orchestration engine. Some call it a process engine. It's all fine. But the idea is that you define workflows which you can run instances of, and then you have all these requirements being settled. I wanted to give you a two minutes demo, not because I want to show the tool, that's a nice side effect, and there are other tools doing the same thing, but I want to get everybody to the same page of what the heck is that, what's a workflow into. And if you want to play around with that yourself, there's a link, it's all on GitHub, so you can just run it yourself. What I use as an example is a small... Oh, that's a nice typo. I didn't see that. It's not an onboarding process. It's an onboarding process. Okay. We fix that later. And then it would be in wrong English, right? It should be an offboarding process. We have a lot of offboarding processes as well. So that's not a bad thing. But it's an onboarding process. We see that in basically every company to some extent. You want to open up a new bank account, you go through an onboarding process as a bank. You want to have a new mobile phone contract, you go to onboarding. If you want to have a new insurance contract, on boring so it's kind of always the same um and this is how it how it could look like and what i'm using here it's called bpmn it's an iso standard how to find those processes you do that graphically i discuss that in a minute as well in the background it's simply an xml file basically describing that it's standardized iso standard so it's that's not a proprietary thing here and then you can do things like okay i scored a customer then i approve the order that's a manual thing um i could and i always like adding things live with the um with the risk of breaking down but we could say that takes too long um we wanna we wanna basically escalate that there we go we don't have to make it nice now we just have to make it work um let's just say escalate yeah we keep it like that and we have to say what too long is so that's a duration with a period time 10 seconds should be enough for a person to review it right 10 seconds so um i just save that um and what i have in the background i have a slide on that in a minute is a java application in this case doesn't have to be java but i'm a java person so it's a javaspring boot application basically that connects to the and workflow engine in this case also deploys the process and then also provides a small web ui and if you know me you know that i do great web uis that's my so i can open your bank account i don't even have to type in data because it doesn't know everything no but i submit the application it triggers a rest call basically and the rest call goes into the spring boot application that starts kicks off a process instance within the workflow engine and then you have tools so i'm using our sa service so you have tools like operate where it can look into what's going on And there you can see that I have processes running. You see the versioning. I have a new version. I have that instance running. If I kick off another one, I get a second one in a second. So I'm currently waiting for approval. And I also already have escalated at the same time. And then you have task list because I'm now working, waiting for a human, for example. I have UI stuff. um i could also do that where chatbot or like teams integration for example yes do automatic processing please so complete the task and then this moves on and i'm seeing that here as well like um as soon as the internet likes me again um i'm seeing that this moves on and also sends an email right so i have that one process instance finish for example um it runs through a couple of steps those couple of steps um then basically connect to either the last two things I want to show for example create customer in CRM system is in this case tied to to a bit of Java code where you can do whatever you want to so that's custom glue code you simply can can program it or if you want to send a neat welcome email you already see that that's a pre-built connector for example for Sendcrit I can simply configure And that means in the background also my email was sent, which I can also show you hopefully here. Whatever. Proof done. Hey, hello, QCAN. An email. Yay. So we're good. Okay. Don't want to show anything else here, I think, for the moment. Just to give you an idea what that is. So that's a workflow engine running in the background. We are having a workflow model. We have instances running through. And we have code attached or UIs attached to either connect to systems or... to the human. Technically, I was using Kamunda as a SaaS service here, and I had a Spring Boot application. If you have any questions, ask me later on. And sometimes I'm being asked, okay, workflow, isn't that for these like, I do 10 approvals a day things? No. So we're having customers running that at a huge scale. So what you can see in that block, which is... oh that was a copy sorry there's a link behind that if you get the slides you get the link for a blog post where we go into the thousands of process instances per second and we run that in geo geographically distributed data centers in the us and uk for example and this adds latency but it doesn't bring throughput down for example and we are also working to uh reduce the latency of certain steps so what i'm trying to say is that that's not only for i run five workflows a day you can run that at a huge scale for core things okay and so far i looked at some business reasons why we want to wait but there are also by the way a lot of technical reasons why you want to wait for things why things get long running and that could be first of all asynchronous communication if you send a message you might not know when you get a message back It might be within seconds in the happy case or milliseconds, but what if not? Then you have to do something. So you have a failure scenario, you don't get a message back. You want to probably just stop where you are and then wait for it to happen. And then probably you can also notify an operator to resolve that. Or the peer service is not available. So especially if you go into microservices or generally distributed systems. the peer might not be available so you probably have to do something about it so you have to wait for that peer to become available that's a problem you should solve right because otherwise yeah i get to that you get chains of um chain reactions basically yeah i'm not going to that and the um the example i always like to use um is this one and I really missed on-site events because I can ask clever questions like who has used an airplane before? Hands. Show of hands. Show of hands. You're still awake. Okay, great. A lot of you. I missed that in virtual settings, really. But then you know the process, right? If you use an airplane, you get an email invitation to check in like a day before. four hours before that normally and then you click a link and you should check in right i did that for a flight actually to london um i think that was pre-pandemic i agree 2019 or something like that i flew to london with euro wings i wanted to check in and what it said to me is the there was an error while sending you your boarding pass so i couldn't check in that's it what would you do try it again yes of course i tried again that's what i did I tried it again five minutes later, didn't work. What was the next thing I did? Call, yeah, that's the typical answer. I come back to that in a second. No, I don't call airlines. I would never do that. I made a calendar entry in my Outlook to remind me of trying it again in a couple of hours. Because it was still time, I mean, it was in the next day. I just wanted to make sure not to forget to check in. And that's what I call a stateful retry. I want to retry it, but in a long-running form, like four hours from now, because it actually doesn't work. But it doesn't matter because I don't need it yet now. So the situation I envision is that in the background they had their web interface, they probably had to check in, let's say, microservice, and they probably had some components like downstream required for that to work. For example, a barcode generation or document output management or whatever. And one of those components, that's my guess, did fail. The barcode generation, for example, didn't work. So they couldn't check me in. The thing is, and I think that's not news. I think that should be hopefully consensus in this room. But the thing is that the more we distribute our systems into a lot of smaller services, the more we have to accept that certain part... parts are always broken or that network to certain parts are always broken. That's the whole resiliency thing we're discussing about. The only thing that we have to make sure, which is really important, that it doesn't bring down our whole system. Like in other words, just that the 3D barcode generation, which is probably needed for my PDF boarding pass I need to print out later, is not working, shouldn't prevent my check-in. That's a bad design, right? That's not resilient. And that means because then you get a chain reaction here. The barcode generation is not working. Hey, probably not a big deal. But it gets to a big deal because nobody can check anymore. And they make it my problem, right? They transport the error all the way up to me for me to resolve because I'm the last one in the chain. Everybody throws the hot potato once further. I'm the last part in the chain as a user. And that makes me be responsible for the Outlook entry. The funny part about that story was, was really the onwards flight, same trip from London. Easy chat, we're sorry. But same problem, I couldn't check in, but they give you the work instruction. They're better with that. We're having some technical difficulties. Okay, got that. Log on again, retry. right if that doesn't work please try again in five minutes i like that increase the interval that makes a lot of sense you could automate that probably right um and next thing and i love that we do actively monitor our site we'll be working to solve the issue there's no need to call it's your problem don't leave us alone right so i'm exaggerating a little bit but this is i mean in this case it's very obvious because it's facing the user but it's kind of an attitude i'm seeing in a lot of organizations even internally to other services. Their problem, which is throw an error, we're good. The much better situation would be the check-in should probably handle that. They should check me in. They could say, hey, you're checked in, but we can issue the boarding pass right now. We're sorry, but we send it on time. Or you get it in the app anyway. I don't want to print it out. Don't need a PDF. So they could handle it in a much more local scope. And that's a better design. It gives you a better design system. The responsibilities are much cleaner defined. But the thing, and I'm coming back to that two or three times later on again, the thing is now you need long-running capabilities within the check-in service. If you don't have them, that's why a lot of teams are re-throwing the arrow. Otherwise, we have to keep state. We want to be stateless. And that's the other part, which I was discussing with a lot of customers over the last... five years I would say, but the customer wants a synchronous response. They want to see a response in the website where it says you're checked in, here's your boarding pass, here's the PDF and whatever. We need that. People are used to that experience. I wouldn't say so. So if my decision as the customer is either I get a synchronous arrow message and have to retry myself or I get Some result later on I know what I pick and it's still better customer experience it It needs a little bit of rethink, but I find it important Let's extend the example a little bit and go into like add a bit more flavor on a couple of those things So let's say you're still doing flight bookings, but maybe you also want to collect payments for it. I would make sense as a company Okay And the payments might need credit card handling so they want to take money from the credit card let's look at that the same thing by the way could happen so you want to want to charge the credit card the credit card service at least internally but maybe also on that level will be or will use some sas whatever service in the internet you will probably not do credit card handling yourself unless you're very big but normally you use some kind of whatever um stripe like mechanism to do that so you will call an external api rest typically to make the credit card charge then you have that availability thing so that service might not be available when you want to charge credit card right so you probably also then have the same thing you want to want to charge it and want to probably wait for availability of the credit card service because you don't want to tell your customers we can't book your flight because our credit card service is currently not available you probably want to find other ways but that's not where it stops and normally then go beyond that which is very interesting if you look into all the corner cases so let's say you give up after some time which makes sense right you don't want to try to book the flight for tomorrow for the next 48 hours doesn't make sense you give up at some point in time and you probably say the payment failed and we probably can't book your flight or whatever it is you do There's one interesting thing about distributed systems. If you do a remote call and you get an exception out of that, you can't differentiate those three situations. You could have reached, probably the network was broken. You have not reached the service provider. Maybe the network was great, but the service provider, the threat exploded while you were doing it. It didn't process it. Did it commit its transaction or not? You have no idea. Or everything worked fine and the response got lost in the network. You can't know what just happened. And that makes it hard in that scenario because even if you get an exception, you might have charged a credit card. It might be a corner case, but it's possible. And depending on what you do, you might not want to ignore it. Maybe you can. If that's a conscious decision, that's fine. Maybe you can't. Then you have to do something about that. And then either you also can do that in kind of a workflow way. You could also run monthly conciliation jobs. Probably also... good solution it always depends right what we heard in the morning and then you but if you want to do it in a workflow way you might even have to check if it was charged and refund so it gets more complicated that's what i'm trying to say okay good one thing um in order to do these kind of things again embrace asynchronous thinking right make an api that's ready to probably not deliver a synchronous result but saying okay we try our best maybe you get something in in a good case but maybe you don't and then that's http codes i like to think in http codes like 202 means um we we got your access we got your request that's the feedback loop we got it but the result will be later and now you can make it long running and that extends your options what it can do speaking of that one of the core thoughts there is also if you make apis like that make it asynchronous, make it be able to handle long running. Within your services, you're more free to implement requirements the way you want. So let's say you extend the whole payment thing, not only to credit cards, but probably to also have customer credits on their account. Some companies allow that. If you return goods, for example, you get it like credits on your account, which you can use for other things. Or PayPal has that. If you get money sent via PayPal, it's on your PayPal account. You can use that first before they deduct it from your bank account, for example. And then you could add that where you say, okay, I first deduct credit and then I charge the credit card. And you get more options of doing that also long running. And... to be mindful of time but as a teaser I'm not going into that today but as a teaser that poses interesting new problems around really consistency so for example now we have a situation where we talk to different services probably for credit handling or for credit card charging all of them have their transactions internally probably but you don't have a technical transaction spawning all of those steps where you say oh if the credit card charging fails i also didn't deduct the customer credit i just say payment failed so i need to think about these scenarios where a deducted customer credit credit card charge doesn't work i want to fail the payment then i have to basically rebook the customer credit and that's for example also what you can do with these kind of workflows where you say that's called compensation i think i marked it yes where you say I have compensating like undo activities for activities if something failed. So it gets what I'm trying to, the only thing I'm trying to say here, it gets more complex very quickly if you think about all the implications of distributed systems here. Okay, good. Yeah, if you want to see code, I'm not showing that today, but there is some code also if you're more thinking in codes than in slides. Okay, but going back to the long-running capabilities. My view on that is you need long-running capabilities to design good services, good service boundaries. So that's a technical capability you should have in your architecture. And I make another example to probably also make it easier to grasp. Let's say the booking service basically tells the payment service via method, via REST call, I don't care, saying, hey, retrieve payment. I don't discuss orchestration versus choreography today because that could be something you're also interested in. Why doesn't it just emit an event? If you're interested in that, ask me later. Happy to talk another hour about that. But for the moment, I don't care. So Bookings has payment, hey, retrieve payment for that flight, for example. Payment charges to credit card. And now let's say the credit card is rejected. Service is available, but they're... credit card is rejected that very often happens in scenarios where i store the credit card in my profile it's expired and then it gets rejected now the next question is what to do with that and typically a requirement could be if the credit card is rejected the customer can provide new details they hopefully still book their flight right we want them to do that but they need to provide more new credit card details you can also think about other scenarios i i somewhere i have the example of github github subscriptions because there it's even a fully automated process that renews my subscription uses my credit card it doesn't work they send you an email update your credit card the question is where to implement that requirement and one of the typical let's say reactions i'm seeing in a lot of scenarios is that we're as a payment we're stateless again we want to be simple we we can't do that because then we have to send the customer an email we have to wait for the customer to update the credit card details we have to control that whole process it gets long running they're not normally not using that word but it gets they understand it adds complexity they don't want to do that so just hot potato throw to the booking because the booking is long running anyway for for a couple of reasons they also have that they can handle that requirement better so let's just throw it over the fence over there i'm seeing that very often actually if you make the same example with order fulfillment or other things where it's very clear that that component like booking order fulfillment has kind of a workflow mechanism then this happens the problem is now you're leaking a lot of domain like concepts out of payment into booking because booking shouldn't know about credit card at all. They want to get money, they want to have the payment. They shouldn't care about the way of payment because that probably also changes over time and you don't always want to adjust the booking just because there's a new payment method. It's a better design to separate that. So that's questionable. If you go into DDD, for example, it also leaks domain language like credit card rejected. I don't care. I wanted to retrieve payment. Either you got my payment or you didn't. That's the two results I care about as booking. So you want to really put it into the payment service. That makes more sense and then get a proper response, like kind of the final thing. But in order to do that, you have to deal with long-running requirements within payment. And that's the thing. You should make that easy for the teams to do that. All right, small burning thing here, but I added potentially, or no brackets, it's not that sign, it's that sign. Potentially on the slide. In such a situation, payment might be, in 99% of the cases might be really super fast and could be synchronous, but then there are all these cases where it might not be, and it's good to be able to handle them. And then you can still... design for example an API where you say in the happy case I get a synchronous result but it's kind of an a not an exceptional case but it's it's just one case the other case could be I don't get that I get an HTTP 202 and an asynchronous response make your architecture ready for that good and then you could use surprise surprise probably also workflows for implementing that and by the way just because there's a workflow orchestration doesn't mean it's a monolithic thing That's something I discussed with Sam Newman a couple of times in the past. I would even say the other way around, if you have long running capabilities available in the different services you might want to do, it gets easier to put the right parts of the process in the right microservices, for example, and it's not monolithic at all. It gets monolithic if... for example payment doesn't have long-running capabilities and you move that logic into the booking service just because that booking service has the possibility to do too long running so it's not i'm not sure if i'm inviting too much on that but i find that important it's not that the having orchestration or long-running capabilities adds the monolithic thing it's the other way round because not all the services have them at their disposal normally, what they do is they push all the long running stuff towards that one service that does and then this gets monolithic. Okay, you still with me? Okay, good. So from my perspective, having long running at the disposal for everybody avoids these what Sam Newman one called God Services. Okay, so long running capabilities are essential. It makes it easier to distribute all your responsibilities correctly and also it makes it easier to embrace long-running asynchronous non-blocking stuff. Good. Or in my other words, and this, by the way, it's me in front of a tent. That was at CraftCon. That's one of the venues where you do the talks. I'm not sure if you can recommend different conferences here. I think I can. so also very much recommendable so you need a process orchestration capability that's what i'm convinced of i mean otherwise probably i wouldn't do it for all my life but um but that's also easy to get as a team so nowadays that means as a service either internally or probably also externally to create a good architecture i'm really convinced by that okay looking into that how can i do that how can i get that into the organization better is one thing we're what we are seeing very successful all let's say all organizations i talk with that that use process orchestration to a bigger extent very successfully they have some center of excellence like organizationally they not always call it center of excellence sometimes it's a i heard digital enabler or i even heard process ninjas once so it might be named very differently that depends a little bit on company culture and things but it's a dedicated team within the organization that cares about let's say long running if you like phrase it more technically or process orchestration process automation these kind of things that's by the way is a link for a 40 pages article where we collected a lot of the information about centers excellence how to build them how what are best practices to design them and so on so forth So if you're interested, that's probably something to look at. And one of the core ideas there is a center of excellence should focus on enablement and probably providing a platform. They should not build solutions. Because sometimes people ask me, OK, but we did that BPM and so on. Somebody remembers BPM and so on. Not that many people. OK, good. Either the others are good in deleting memory or young. Probably both. where we had these kind of central teams doing an esp and very complicated technology and didn't work it didn't work because at that time a lot of those central teams had to be involved in the solution creation they had to build workflows they it was not possible without them and that's a very different model nowadays you probably normally have a central team that focuses on enabling others that then build the things and enabling means probably consulting helping them building kind of a community, but also providing technology where they can do that. And what I'm discussing very often, let's say within the last two or three years is, oh, but we stopped doing central things. We want to be more autonomous. We have teams, they should be free in their decisions. We don't want to put too much guardrails on them. Isn't the central COE kind of the path? Why do you do that? I discussed that with a lot of organizations actually and that's something also the team where I was so happy about the team topologies book and there are a couple of talks here also looking into team topologies that's definitely a recommendation to look into and it's also the core ideas are very crisp actually and the core ideas are those you in order to to be very efficient in your development you have different types of teams So that's the stream aligned team that does business logic, that implements business logic basically. They provide the value if you want to say so. That's very often also value streams and whatever. So you want to make them as productive as possible to remove as much friction as possible so they can really provide value, provide features if you will. And in order to do that, you have other types of teams. The two important ones are the enabling team. kind of a consulting function like hopping through the different projects and the platform team providing all the technology they need so they don't have to figure out everything themselves the complicated subsystem team is something we don't focus on too much it can be some whatever some fraud check AI thing somebody does and then provides an internal as a service thing whatever okay and you can map that very well and our customers do that actually very well to having a center of excellence around process orchestration automation for example where say they provide the technology in our case that's very often commona but it could be something else very often they also own by the way adjacent tools like rpa tools robotic process automation and others they provide the technology and also the enablement project templates and whatnot and that's very efficient actually And it frees the teams of figuring out that themselves. Because that's so hard. As a team, if you have no guard, if you don't have an idea how you build your stack, you can go into evaluation mode for two or three months. And you don't deliver any business value there. And that's actually not new. And there are a couple of good or recommendable blog posts out there also talking about that. One is... um the uh thing from spotify spotify um published about golden path uh 2020 i think is it there yeah 2020 um where they basically said okay we want to want to have certain um defined ways of building a certain solution type if we build a customer facing web app this is normally how we do it if we build a long-running workflow this is how we do it right so they have these kind of solution templates the name is good actually they name it golden path because it's golden they make it so easy to be used so they don't force teams to use it that's the autonomy thing they don't force it upon people but they make it desirable to be used to make it here it's easy and by the way it's not your fault if it's not working that's kind of the thing right and then it's golden and they are and i like the blog post actually they uh i love that quote because they found that rumor driven development simply wasn't scalable i heard they do it like that probably should do that as well right and then you end up with a with a quite a zoo of technology that doesn't work so i find this really really important that You want to consolidate on certain technologies. You want to make it easy to use them across the whole organization. That makes you efficient, but don't force it upon the people. And they also have a tool. I mean, it's in the company. They do open source on the side. They have made backstage. I'm not sure if you heard about that. It's the link down there. I have no idea if the tool is good. I have not used it at all. But I love the starting page of their website. where they said the speed paradox at Spotify, we're always believed in the speed and ingenuity that comes from having autonomous development teams. But as we learned firsthand, the faster you grow, the more fragmented and complex your software ecosystems become, and then everything slows down again. The standards paradox by centralizing services and standardizing your tooling backstage to enhance your development environment, instead of restricting autonomy, standardization frees your engineers from infrastructure complexity. And I think that's an important thought. And they're not alone. So if you search the internet, you find a couple of other places, for example, Twilio, but also others. Same thing, right? We're offering paved path, they call it paved path, mature services, pull off the shelf, get up and running super quickly, blah, blah, blah. And what you do is create the incentive structure for teams to take the paved path because it's a lot easier. if they really have to go a different route you make it possible so it's not restricting autonomy simply helping them and that's important and i think it's also important to discuss that internally anyway second thing um last thing um graphical models that's the other thing i discuss regularly okay okay center of excellence yeah probably makes sense process orchestrating yes i understand why we have to do that but graphical models we're developers write code um thing is bpmn that's what i showed it's an iso standard it's worldwide adopted it can do pretty complex things so i just scratched the surface but it can express a lot of complex things in relatively simple models so it's powerful it's living documentation so it's not a picture that's requirement but it's running code that's the model you put into production it's running code and that's so powerful because then you can use that in For example, this is an example where it's used for test cases. That's what the test case tests, for example. You can leverage that as a visual. Or you can use it in operations like where is it stuck or what is the typical way it's going through or where are typical bottlenecks and so on and so forth. And you can use that to discuss that also with different kinds of stakeholders, not only developers, but all of them and even developers. I mean, if you discuss a complex... algorithm like a longer process or workflow you normally go to the whiteboard and sketch it because we are visual as a human just because i'm programmer doesn't make me less visual i want to see it very very powerful and it's even more important because i think a lot of the decisions about long-running behavior needs to be elevated to let's say the business level they need to understand that why we want to get asynchronous, why this might take longer, why we need to change also customer experience to leverage the architecture. And the only way of doing that is to really make it transparent, to make it visual. Another way of phrasing that is one of my favorite pictures is this. In order to leverage, that was, I think it was a former marketing colleague that worked with me, phrased it like that. Okay, what you're trying to say is that in order to leverage your hipster architecture you need to redesign the customer journey and that's exactly that right so that's important to keep in mind i want to quickly close that with another flight story the last one i promise i just have a minute left so um and that was yesterday i know the first thing it's happening by the way the first thing it's happening so you get everything asynchronous so they did change the customer experience a lot now i'm working on train companies so it's the same thing so but anyway mobile you get automatically checked in for flights you don't even have to do that why should i do that but um and yesterday that was a fun part yesterday um my flights to london seem not to be my lucky bet actually so my flight to london yesterday was delayed by an hour okay that's delayed that was cancelled okay that's not so nice Then I got a relatively quickly an automated email. That's the only one in German, which I don't get. Why did I get that one in German? I don't know. But it was in German. Okay, I can read German. That says for, I got the link to book my hotel at Frankfurt. Why? I don't want to get a hotel. I want to get to London. But everything automated, everything pushed. Nice. Then I got a via the app, not via email, a link to a chat bot where I should. chat about my flight and it says okay we rebooked you for tomorrow morning okay it didn't do that completely by the way because it's not lufthansa so you have to see a human colleague in order to first like okay but i don't want to get to london tomorrow i want to get there today so um i basically visit the counter and the end of the story is they could rebook me to a very late flight to london heathrow which was very late and london he throws i hated that but what i what i still like everything was actually gonna sneak they and i got like notification of everything in the app via email so i think there are some some uh some good things on the horizon there so the customer experience for for airlines at least changed quite a bit over the last five years i would say um funny enough last anecdote i i wrote a i wrote i read an article about the bad nps score of lufthansa on this weekend and i probably understand why recap you need long-running capabilities for a lot of reasons and process orchestration platforms workflow engines great technology you should definitely use that for those because it allows you to design better service boundaries implement quicker less accidental complexity you can embrace asynchronicity better provide a better customer experience and we haven't even talked about the other stuff like increase operational efficiency automation reduce risk be more compliant document the process and so on in order to do that successfully across the organization you should organize some central enablement and i'm a big advocate for that to really adopt that at scale if you want to learn more there's my book there's website we have our own conference as thank you very much

Watch the InfoQ video with the transcript included: https://bit.ly/48mYS1r 

Key Takeaways:
1. Importance of Long-Running Processes: Understand why handling long-running processes is critical in modern, distributed systems and how it impacts software engineers across various services.

2. Challenges with Distributed Systems: Learn about common challenges, such as remote communication failures, peer unavailability, and time-sensitive business problems in distributed architectures.

3. Role of Process Orchestration: Discover how tools like microservice orchestrators and workflow engines enable efficient handling of long-running processes, even at massive scale.

4. Decision-Making for Domain Boundaries: Get insights into designing effective domain boundaries and making informed architectural decisions with the help of long-running capabilities.

5. Hands-On Learning: Explore real-life examples and access sample code (available on GitHub) to experiment with and implement long-running processes in your own projects.

#SoftwareArchitecture #Orchestration #Infrastructure