Maxim Fateev - Temporal

Conceptually, this is the first technology which abstracts out distribution from developers, right? So because this process is not linked to a specific machine, so we practically do like migration that process seamlessly. You can think about it as building operating system for the world. Hello, welcome to DevTools FM. This is a podcast about developer tools and the people who make them. I'm Andrew, and this is my co-host, Justin. Hey everyone, we're really excited to have Maxim Patif on with us. So Max, you are the CTO and co-founder of Temporal. And a long time ago, we actually had Sean Wang on SWIX, who was at Temporal at the time. So it's been really exciting to follow your development on this like pretty, like kind of really important and ever growing and important product. So we're really excited to have you on and talk about that. But before we dive in and start talking about Temporal, would you like to tell our audience a little bit more about yourself? Yeah. Okay. Thanks a lot for having me. Just a very quick background. Most of my life I worked for large companies and I spent eight and a half years at Amazon, a couple years at Microsoft, four years at Google, and then my last job before Temporal was at Uber. So I kind of be company. I was engineer. I never... had a single report before I started the company. And the first four years and a half, I was a CEO. And now I switched back to CTO role. And I have zero reports right now. So I'm one of those CTOs, which does only technology more like chief architect for the company. So this is kind of my background. And just to give you some idea what I was working before that, at Amazon, I was tech lead for the Amazon messaging platform. which practically was a broker-based architecture for PubSub. Well, Amazon ran on that for a while. And it was years before Kafka was even conceived. And then later, that project was used as a backend for a simple queue service. And then later, I was a tech lead for Amazon simple workflow service. And the simple workflow service was kind of original idea behind what we later implemented as an open source project at Uber. And the project was called Cadence. And later, we started a company and forked the project as Timporal. So Timporal is an MIT licensed open source project and there is a company which monetizes that as well. So what are the sort of problems that you were running into with or that these companies were running into with sort of traditional like background tasks and queues and other things that like you actually needed a workflow engine for like what was the original motivation? Original motivation is that every application and every application developer is distributed application these days, most of them. And we give people kind of low level abstractions and we also give them a bunch of patterns how to assemble those abstractions into something useful. The problem is that as patterns go, you have to reassemble the whole thing every time yourself and every time you do it differently. And there is no kind of way to do it. There's no underlying middle way which will take out of most of that complexity. And the other part of that is, if you think about your business logic, like your business logic can be pretty straightforward, like I don't know, call these 15 APIs in certain order, maybe based on some conditional logic, maybe dynamically. And if you look at that, and then say, Okay, but how do I transform it to the scalable, resilient application, then your logic gets spread out across multiple pieces, a bunch of callbacks, a bunch of services. And then most of your code will be you'll have nothing to do with your business logic. So like 10, 15, 20 lines of these API calls can be transformed to 1000s and 1000s of lines of logic, which has nothing with your business logic. And this is what we're exactly trying to solve is can we go back and abstract out most of the complexity from developers and let developers focus on what they kind of have to do is build applications and build applications which can scale and be resilient. So when I was reading through your blog posts, I saw that you had a talk about building from first principles, the temporal workflow engine. When it comes to these long live workflow things, what are those first principles? I think, OK, just before going back, because you keep calling workflow engine. And I think this is not really good way to position that. Because first, it just probably term workflow engine gives very bad taste for most developers. The moment the developer hears workflow engine, they just don't want to even hear like up to that. I made this mistake making talks at the conferences called workflow engine, whatever, and nobody comes because nobody cares. While the problem we are solving that everybody cares about that problem. And the solution we should propose based on new abstraction, and this abstraction we call durable execution. And what is durable execution? Think about it. Right now, as a developer, your code can crash anytime, right? So practically, most of your complexity comes from the... understanding that your program can just disappear, right, without any notice. So that's why you always need to break into the pieces, you practically need to persist state all the time, right, so you either use database or queue, and you need to reload that state on every callback, and so on and so on. And what durable execution does, it practically says, okay, imagine that you can preserve the full state of your code execution all the time, durably, right. So in the most primitive form, imagine you are Okay, these days we are writing AI workflows, right? Like you write AI-based workflow, you call LLM, LLM returns a list of tools, you call a tool, and then your process crashes. Then at this point, practically, you lost everything, right? You need to go back and recall it. And imagine it was long-running things like, I don't know, let's say OpenAI does research, right? So imagine if you implement something like research yourself, right? If you fail in the middle, it's a longer running operation, it's not good. And if it's temporal, what will happen is that if you call the tool and process crashes, then tool reply can come when it comes back, and it can come back like three days later. The temporal will resurrect the state of the function as it is still blocked on the tool call, right? These are all variables, all the environment present, and then go to deliver the result, go to the next slide. So from the developer point of view, crash didn't happen. So in fact, imagine crash with execution, right? Double execution, crash with execution because cannot crash and it's always persistent. And there are a lot of applications of that because most important implication is that your code can run as long as necessary, right? Because it cannot crash. Then you don't need the database explicitly because you can just store something in a local variable and it's guaranteed to be persistent and durable as long as you need. And then, as you said, an API call can take any amount of time. So you can make RPC, which takes five days, right? Imagine how simple your system is if you need to think about all this, I think, stuff, events going around. You just call an API. Three days later, it returns. From your point of view, there is no difference if it took 50 milliseconds or 30 days. And the most basic stuff can be sleep. Imagine you can, in your code, you can, if you want to implement something like subscription, customer subscription. you can do something like for loop 0 to 12, sleep 30 days, charge send email. Obviously, 30 days should be calculated based on calendar, but conceptually, you get the idea. And you can actually keep people call blocking sleep 30 days in your code. And it actually makes sense in production code, right? Because process cannot crash. And that is, I think, like kind of programming model. And then as you said, that presentation was mostly about backend, how you create scalable backend which supports that model. And we obviously can talk more about that, but it's a separate conversation. Yeah, so help us like piece together the full story here. So Temporal is a durable execution engine and you have these SDKs in different languages where people are implementing their business logic. they have a TypeScript SDK or a Python SDK that's, as you said, like making an AI tool call or something. And in that runtime, in that SDK usage, they have local state, they're making calls to other systems like OpenAI or something. Let's say OpenAI fails, like how does Temporal help at that point? Or what does the user have to do to take advantage of Temporal to make sure that that failure doesn't crash their hole? workflow. So if you make all this logic and all this registration logic, you just run it inside of what you call workflow function, which is durable execution, you so practically the main requirement of this model is it is implemented by temporal is that workflow orchestration function cannot make a direct IO, it should be deterministic. So it means that all LM calls all two calls should be done in activities. We record results of those and we use event sourcing to recover state. And that means that, for example, if you take something like, I don't know, OpenAI Agents Framework, and you can intercept all calls to LLM and all calls to tools, then you can run that framework directly in the workflow code. And then it means that any crash of the process will not affect the workflow state. And activities are retried automatically based on the specified retry policies. So if LLM fails, you just... Again, it can fail with error, right? And if it's a retryable error, it will just retry automatically. It can time out and retry, or it can be error which is non-recoverable, so you will return it back to the framework, to the workflow, and then you'll take care of that based on whatever business logic is. So you have full control of that. But things like automatic retries... and Tupor doesn't limit duration of retry. So if you have a use case when you want to retry for a month, you can retry for months. There is no limit how long an activity can run and how many times it can retry. It's all built into the system. If you're building a DevTools company, you're eventually going to have to support enterprise-grade customers. And that means building SSO, RBAC, Directory Sync, all the things you don't have time for on your roadmap. And that's where WorkOS comes in. It's a modern platform for enterprise-ready auth and access. With just a few lines of code, you can integrate AuthKit, a drop-in login UI that handles sign-in, pass keys, MFA, password resets, and more. If you need directory sync, WorkOS supports SEIM provisioning with all the major providers, so think Okta, Azure AD, Workday, whatever you use, and it keeps users and groups in sync automatically. Their admin portal is a game changer too. It gives your team a self-service dashboard to set up SSO or verify domains or manage access without you having to build that out yourself. And when you're ready for more advanced permissions, WorkWest makes it easy to define complex access rules without building your own policy engine. So if you're scaling your DevTools product into the enterprise, skip the boilerplate and go to workwest.com and get back to building the fun stuff. So Temporal likes to describe itself as a developer-first engine. So what does that mean in practice? What are all the different ways that I can run it and utilize it in my business? So think about it. Before durable execution was a concept, what we called workflow engines, right, existing ones, they were trying to practically... You had a choice, right? You practically said, now you write your code, like, for example, in Java, Go, Python, TypeScript. But it's not durable. If you want to make it durable, now you practically need to convert your code to a JSON document or UI kind of pictures, right? Or DAG or graph or whatever. So practically you always had to create this kind of intermediate representation. And then we end up with things like, I don't know, BPMN or step functions when people practically try to implement high level languages as these syntax trees, right? Like using JSON or XML. And these attempts are very developer unfriendly. And there is this promise that non-technical people will write code, but the reality is that after a certain complexity, even developers hate that because it's very hard to replicate power of Java or Go or TypeScript in JSON, right, or XML. And Temporal practically says, no, no, you actually, because you have durable execution, you just write normal Java code, right? You have full power of the language. You can use classes, you can use interfaces, you can use ICN code, you can practice. There are a lot of things you can do. Practically, it's full power, even reflection. And it applies to all other languages as well. So you practice state within your ecosystem, use preferred language, we support A7 SDKs and the different languages. And you can actually do cross-language flows as well. And then you can use the same unit testing frameworks. If you're in Java, you can use JUnit. You can use your CI CD pipeline. It's just part of your code base. And you link into your application. You don't need to ship this code somewhere. It's part of your application. It's part of your service. All you need is connection to a backend cluster which keeps safe, like Temporal service. And this Temporal service runs on top of a database. Open source runs on top of MySQL, Postgres, Cassandra, and SQLite. Or you can just point that to Temporal Cloud backend, and the same code will run without changes, everything your service, and you deploy your service, you manage your service, which is good from security point of view, by the way, right? Because that's why a lot of enterprises, very impressive about user data a lot, use T-Borall because they can, they run code themselves and encrypt everything before sending it to T-Borall backend cloud, for example, when they use a cloud. So there is no need to look at the data, this pass through. So that's why we pass security reviews of most security conscious companies. Oh, that's interesting. So your event log that you're using to back the durable execution doesn't actually have to have those strict values. You can just have like some encrypted log entry. And so long as I can be decrypted on the client for the SDK, then you can use that cloud persistence without actually having to see like what the data is. Exactly. Exactly. And I think if you go to security, there are three main reasons how I explain people why you can use our cloud versus self-hosting open source. is that first you run the code. So cloud doesn't know what your code. Second is you encrypt everything before sending it to the cloud. So with your keys and your encryption algorithm. And third one, you only connect to the cloud, and it never connects back. So then you open any holes in your firewall. And obviously, most enterprise use private link even to connect to the cloud. So it's kind of secure every security person talks to us and okay, it makes sense. So help me understand a little bit. So you're integrating Temporal into your backend system. You're running these workflow functions to make a part of your business logic durable. Is Temporal itself actually kicking off a call to that? Does your system have to say, okay, I want to spawn this workflow, and then it just happens behind the scenes? I'm trying to get a good understanding of like the execution model. It's like who is actually calling these workflow functions and like when does that happen? So let's think about, let's say we just take very simple agentic application, right? Like you, let's say you start, you make a call, LLM needs to, let's say, make a call, couple tools and return the results. Like let's say we don't do a loop, we do a single, simple one, right? So first you will go and let's say it relates to some, I don't know, inquiry, right? Like some order ID. So you will go to SDK and call, say, Start Workflow Execution. And this SDK will call into the temporal gRPC interface with the Start Workflow Execution gRPC call, passing all inputs in encrypted form, right? And also specifying Workflow ID. Temporal is a fully consistent system, so it guarantees uniqueness by ID, business ID. In workflow ID, in this case, probably will be order ID, right? Maybe order ID dash inquiry, whatever. And then this task, we practically will update state inside of Temporal Service. And as I said, it can be self-hosted, open source, MIT licensed Temporal Service, it can be Temporal Cloud, it doesn't matter. And then this service will do two things. First, it will store the state, and second, it will create a task. And this task will be put in a special queue, like a practically workflow task queue. And this task will be picked up by a worker. You host the worker. Part of your SDK, like your application, will be what we call worker process, which is practically a listener to that queue. So it listens to queue and we'll get the task. The task will say, oh, we just started this workflow with this type and this ID and these arguments. And then this case will be, okay, this is an AI agent workflow. So it will find the code for that, start executing that. And then execution will say, I need to call LLM. But LLM is an external call, so it cannot be done directly. It should be done for a command. So for an activity, so this code will just generate a command, schedule activity task, with inputs and all data related. And again, encrypted will be sent back to the service. Service will update the state, create a history of events. and then create new task for activity task queue. An activity worker will pick it up, execute the LLM call, put results back and deliver, say complete activity task back on the server jar PC interface, which will create workflow task workflow will pick it up. And now it says, okay, this is done, right? So I can pull next thing, which will be a tool and create new command to execute a tool. It will create new task for activity. So it kind of is dense between So we practically never call these things directly. We always call them through queues. This is all fully asynchronous and event-driven on the back end. So you get all the benefits of an event-driven system. So you get backlogs, you get rate limiting, you have flow control, and your workers don't need to open ports because they pull long-pull for these tasks from the queue. But from your point of view, make an RPC call. And this RPC call returns pretty fast. But again, it works very well, even if things are overloaded, or like there's a backlog, so you get all the benefits of that. And then at some point, after everything is done, the workflow will say, return the function, return the result, it will practically make a command, complete workflow execution back to the server. And then at this point, the client might be starting workflow asynchronous, so it can be actually waiting on the result using Longpole behind the scenes. And then this moment client will unblock and say, here's the result. So kind of there is this dense server, worker, server, worker, and then back to the client. Yeah, I think just one thing I want to kind of just double click on. This architecture is extremely robust. Any component can go down and anytime and you can't bring it back because again, we have a worker listening in the queue, right? Every task has a timeout. So if it crosses the task and goes down, timeout will kick in, queue already delivered that task, right? So if your workflow is like a state is lost, it will recover the state using event sourcing and then continue from where it was left off. From your engineering point of view, you can add hosts, remove hosts, add capacity, remove capacity, and the process can go down and back. Everything will just continue from where it was left off automatically without you doing any special treatment. I think you preempted my question there because I was going to say, there's a lot of coordination going on here. There must be some operational complexity to making sure this is robust. But if you have it where it's like the components have good fault-tolerant boundaries, then it sounds like it's not as bad. In practice, how many services actually are running in total? Is it just the two main services? You mean the backend? Yeah, on the backend. No, backend is... One, two, three, four. Practically frontend, which is stateless frontend, which practically does routing. Very so-called history service, which practically maintains the state and events and all the kind of state machine of business logic on the backend. And then there is matching engine, which practically keeps these task queues and also performs like supports long poles, right? Because we support long pole and matching long poles to requests, requires special kind of treatment there. A nice thing about task queues is that they are fully dynamic. You can have unlimited number, practically unlimited number of them. So it's very common to have task queue per process, for example, if you want to route tasks to specific processes, right? So you can, if you need to, for example, say first task hits any host, but second, third, and fifth should go to exactly that host because data is cached there. We can easily support that. And also one more role is worker is practically a background process. And we have our own workflows which run on the background doing all sorts of interesting kind of maintenance tasks. So we call it worker role. So you've been at this workflow game for quite a while now, both before Temporal and now. What are some of the most surprising ways you've seen developers use Temporal? Okay, nothing surprises me really, because it's a very, very general purpose platform, right? Practically, it just makes you code robust. Every time you need a grantee of execution. you will do that. So it's practically used everywhere. It's like we... you can practically start from like up the stack. For example, at Uber, you need to deploy a new version of a kernel to the data center. How do you do that? You deploy it and then you need to reboot every machine. So coordinated reboot of every machine in your data center, all the hundreds of thousands of machines, obviously, it's a workflow using this technology, right? Then you go up the stack, then you need to manage deployments, right? So you need to manage your Kubernetes clusters, you need to manage your infrastructure. For example, HashiCorp, their cloud service is built around this technology because it allows them to coordinate, practically deployments and the maintenance of the backend of their, like, I don't know, NMAT clusters and other resources. Datadoc uses that to practically do all the backend orchestration. That is kind of another one. for kind of this and a lot of companies building control planes. If you need to build control plane for your service, that is one of the best technologies to do that. Then you go up the stack, for example, you want to do application deployments, right? CI CD pipelines that Netflix, for example, rebuilt their version of Spinnaker on top of that. And now they decided not to even use Kubernetes at all because it stopped scaling for them and they just use the temporal directly to practically do all the infrastructure automation. Then you go up the stack and you talk about data pipelines, right? You talk data pipelines, data movement, like everything related to processing data. Because think about most data solutions, they are good crunching files. But the moment you say, oh, when I'm processing this, I need to do a bunch of API calls, which are not reliable, data pipelines, usual data pipelines technologies are not good for that. And for example, like you're doing invoice generation, right? Or doing end of months invoice generation. You have a big, big job, but every invoice can be one API call. It can be 10,000 API calls. For example, at Uber, when you had invoices for companies, some companies are huge, right? So invoice can take one hour to generate for one company, but it can be small because it's a two-people company. So how do you orchestrate those? And then you go up the stack of things like payments, right? So a lot of banks and companies, like every Coinbase transaction goes for this type of system, right? because they need reliability of transferring data between different blockchains. So payments, real-time payments in India, first of all, in Brazil, like you know, because UPI in India, right? This type of instant payments, people use it for that. And then like a real back-end payments, like banks in your orchestration between each other. And then you use things like customer onboarding, business flows, like the norm. And now AI agents. AI agents is a very powerful use case there that people use it for. So because if you think about it, we focused a lot on frameworks, right? How you actually could make these calls and how you transform data. But we have so many frameworks right now. Why? Because it's not that hard to write a framework. But the problem is that running this framework at scale, resiliently, with all the failure modes is actually much harder. And what we're seeing right now that a lot of people who know about temporal, they directly come to us and say, can we just integrate all these frameworks and make them run resiliently? So this is what we are looking for to do is that can we just make the existing frameworks and run them on top of temporal so you get the benefits of all the nice abstractions they're creating and we make sure that we can run these things at scale and resiliently. It's really amazing how broad of a use case it has. But, you know, durable execution, I think is it's probably one of the more powerful concepts. I mean, even though like it's sort of like it's existed in other forums for a while, it's one of the more powerful concepts that I think has been coined recently. And there's a lot of startups in this in this space that I think Temporal has like inspired to tackle this problem more broadly. um i actually worked at one myself for a little bit and i wanted to ask you about like a technical problem that we like we had ran into so you mentioned previously that you're using event sourcing for um like capturing essentially like side effects and making sure that those are logs so they don't have to be re-ran and you can just use old results one of the challenges with that approach is if you have uh either really large payloads you like you get a really large response back. Or if you have like a streaming response, then it can be hard to store those effectively in the logs. So like, how does Temporal try to help with these? And I think like the streaming response is like really applicable to maybe AI workflows. I guess they're doing a lot of those. So, but I want multiple questions there, right? One is large payloads. There are two really different use cases for large payloads. One is that you just need to pass it through from one activity to another, right? You need to look into that, you just got file, you pass this file to another activity. So usually what you do is you do some level of indirection, right? You put it in some other storage, for example, S3 or some blog store or some even a local file, we actually can because we can route tasks to the same host, right? You can practically cache it locally or you're in process. And that is one way to solve that efficiently, just local caching and maybe storing and then point when you're passing pointers around. And we are working on framework level features to make this passing pointers around seamless, like in EZ. That is one. Another one, sometimes you want to look into the payload inside of the workflow code. And there, there are a couple options. One option is that you just, again, cache these things and you return them very fast. And second one is sometimes you don't need the full payload to recover workflow state, because usually, imagine you have a function which counts words in the file. Right? So all you can... and then you say, oh, if like, if I have less than 500 words, take this path, otherwise take other path. The only thing you need to store is the counter, right? Not the whole file. So there are techniques to make that efficient. So you could be able to kind of only store smaller results. And we will provide them as well. For streaming, it's an interesting question. Do you know any use case for streaming in the AI world, which doesn't involve human waiting for tokens? Not really, not really. I mean, it's just used as a sign of progress mostly. Yeah. So, you know, I'm old enough to remember dial up. And I remember that I acutely knew how picture every JPEG loaded, right? It could load from the top, it could load interlaced, right? Like it was like the whole thing watching the JPEG loading. So my claim is that we are living in the JPEG like dial up vault of LLMs. And the moment LLMs become fast and produce those tokens faster, the whole streaming will go away because nobody would care about that because it just will appear instantaneously. And all backend agents, like agents which are actually doing real work on the backend, for example, calling tools and doing other things, they don't need streaming. You need the full result to actually make your next decision. So reality is that workflows don't need streaming. There are some use cases when streaming is useful and we are going to integrate them. I have ideas how to do that. I probably don't have time to kind of go in details there. But the reality is that for all practical reasons, streaming is purely entertainment for the user while it's waiting for your kind of LLM to do the work. And you don't need workflow engine for that. You can just pass the stream directly. You can use Temporal to pass the pointer to the stream to the client efficiently. and then you can we can support streams naturally you can use something like radius or some other technology but i don't think there is a really need for durable execution to support streams for these specific llm based use cases yeah that makes a lot of sense um what other like kind of tricky problem i actually just got back from the local first conference and this is a big topic um is just dealing with like versioning of workflows And a few years ago, I actually worked at Oxide Computer Company and they have this same sort of thing because they have a control plane. And when your data model is evolving, so let's say the workflow is evolving and you've got these in-flight streams or maybe a service has crashed and it's in the event log, but things have changed now. How do you deal with that versioning conundrum? What does that look like? So first up, before even going how we solve that, we should realize that an alternative is that ad hoc system which doesn't even have story around versioning, right? Like, okay, you have these 50 components, multiple databases, multiple schemas, multiple services, and now try to move that ship in the right direction. With a durable execution, you have very explicit ways to deal with versioning, right? And there is very clear story. And obviously, it's not like, you cannot hide versioning because if you have a process which runs for months, and you want to change it while in flight, you cannot just make it absolutely seamless. It's not going to work like that. So there are two really ways to do versioning with these type of systems. One is when system is workflow is short lived, let's say it lives up to a certain amount, like a few minutes, maybe a few hours, maybe a day, you can run multiple, practically you pin the version of the workflow to a specific version of code. Right? You practice say, I start this workflow in version one, and I have set of processes which support version one set of workers. And I will keep this set of processes around for as long as workflows with this version are open. So it works pretty nicely if you have short workflows, because again, you don't need to run too many of those. We call it a rainbow deployment. So your build system needs to a deployment system, you need to support running multiple versions at the same time. This Kubernetes is not that hard, right? You can just run them. Another approach which we don't have support right now, you technically can load multiple versions of the same kind of code in the same process in some languages. So you certainly can do it in TypeScript, you certainly can do it in Java with class loaders. Maybe you can do it in something like in Python, but then spawn in separate, like you can walk around these things. So we will provide more of that. And that is another approach. And then for long run process, it still doesn't help, right? Because if you have a process that runs for months, you have a bug at the end, you still didn't reach that point, what do you do? And we support that as well. And the way you do that, you just, it's pretty actually simple, but it's not perfect. You just keep both versions of the code inside of your code. You practically have kind of conditional statement. You say, if old version, this is old code. If new version, this is new code. And this allows you, if code... it didn't reach that version, it will take New Code Pub. It's already passed that old code for recovery, it will use old one. So you practically end up doing that. And then you have tools and processes how to validate that this is correct. So you can do replay testing. And practically, you can just go and say, for them, you can download a bunch of histories, check them in into your... repo and every time you for them run tests, you can go and replay all your workflows against old histories and make sure you didn't break them by mistake. That is absolutely recommend everyone using the technology doing that. Also, we are working on safe deploy features. So we will do like self rollout, self rollbacks, and this like testing of replaying before actually doing migration. So we will add more features around safety of deployment just to make sure that If you by mistake, you don't, for example, put the correct logic there, we will catch it before it messes up your production. Yeah, I think that's one superpower that we really haven't talked too much about, but like the ability to replay and the test replays. This is the hardest thing in distributed systems. It's like you have a bunch of components and they all have their own behavior and like figuring out how they'll integrate can be really, really challenging. So being able to replay things. I think just generally, I think if people ask us, we ask people why you, use temporal and why you like this durable execution approach. First, just less code and nicer code. You need to think about it differently. But second always comes up, sometimes first is visibility, runtime visibility. So this is so much because we record every interaction, right? You go and you see every activity call, you see every input, every output. You see every number of every error which happened there. So you practically, if something is stuck, you can go to UI and see that. So the general trapezole, like visibility to the system at runtime and ability to download history and replay under debugger if it failed once in production, it's very, very powerful. So I think, yes, general visibility into your system. Obviously we integrate with metrics, with Datadog, with Grafana. So you have all the other context propagation, so you get all the other visibility, but just general like visibility to your system for like basic UI and history is tremendous. People love that. Yeah, that's awesome. I want to ask one more technical question before kind of we move on and talk a little bit about like open source and open source business models. But an important part of the temporal experience, building these durable execution engines is to be able to access uh or a write code in the language which your application is written in and you'll have a pretty extensive set of sdks so go java python type script.net php ruby like you're covering a lot especially like the web world basis there um but like what is what does it require to support a new sdk is there anything special that a language has to exhibit or is it just like a matter of you know porting like internal apis and stuff So the main requirement for workflow code is to be deterministic. Deterministic means that this code, every time you execute that, you need to end up in the same state, right? So unless you fully own the runtime, and it's possible, something like WebAssembly can do that. If WebAssembly was more mature framework with more languages, we would support it by now. It's just that most people don't use it in production. Every language, we need to figure out how we are going to execute code deterministically. For languages like Python, it's pretty straightforward because we can just get AsyncIO library and write our own practice dispatcher, which will practically AsyncIO loop, which is fully deterministic and runs in exactly the same order every time it's used. For languages like TypeScript, we actually went further. We implemented, we used V8 isolates. So we actually provide fully deterministic runtime container for TypeScript. So you cannot even make non-deterministic mistakes in your code because every API is deterministic. So if you call time, on replay will return exactly the same time. If you call random, on replay will return exactly the same random number. So you're kind of pretty safe. For language like Java and Go, it's harder because they're multi-threaded. We want to support multi-threaded code. So we had to practically write our own dispatcher, which will dispatch your friends one by one in exact order. And from your point of view, it looks like multi-freaded code. In practice, it is cooperative multi-freading controlled by our framework. Unfortunately, it means that in Java and Go, you have to use our APIs for multi-freading. You cannot just say new thread in Java, you practically need to use our sync API. The same thing in Go, you need to write like workflow.go to create GoRoutine. You cannot just do Go directly. Otherwise, you get full power of Go and Java, just you need to use APIs for multi-freading. But yeah, it's a challenge. And every new language is a challenge, and this is the main requirement. And then there is pretty complex backend state machine. That's why we wrote Rust-based core library, which implements most of the complexity of the state machine on the client side. And all new SDKs use that library, like Rust library, as an underlying dependency. So practically Python, .NET... what was the other, TypeScript, they all rely on, and Ruby right now, rely on that Rust core library. Unfortunately, we still don't have Rust SDK. A lot of people ask for it. We certainly will have it one day. It's just prioritization issue when we get time. Because implementing every SDK is certainly very, very challenging. One thing I saw is that a lot of our competition, they probably still don't realize how much effort is to be like multi-language. A lot of them just single or two languages. And yes, it's a price we have to pay for meeting developers where they are. Yeah, I think part of what you pointed out is you have to hook so deeply into the runtime for some of these languages that can be a real challenge. Cool. So open source has been a big part of Temporal's journey. I think you said that it started out as an open source project. So how do you guys as a company think about community building and interacting with open source? Okay, first... It started an open source project at Tupi, right? We build it as open source and then we forked it, but forked as open source. It's still MIT licensed open source, both server side and SDKs. And we also guarantee full compatibility between open source and the cloud. So we even support migration, live migration. So if you're running an open source cluster, you can live migrate to the cloud without downtime. So that is pretty powerful feature, but it requires compatibility. So I think it's pretty because sometimes people can assume they're not open source. It's not true, right? They are fully open source. And the only monetized running the backend server is the key libraries always run inside of yours. From monetization point of view, I think the I've heard this phrase and I think I fully support that is that it's very hard, almost impossible to monetize the library. And we have a lot of cases when we have super powerful and super useful and super widely used open source library. And it's like Docker is probably the best example there, right? And it's still not very monetizable, right? If you look at all successful open source companies, they are practically all have backend component, which is required, like how to... like it requires some management to run, right? Like databases, queues like Kafka, and Tuporos kind of both of them together in a single package. And reality is that... Like one thing about temporal is temporal is by just durable execution temporal in general, always end up in the mission critical path of your company. Because reality is that this is the better programming model. And once developers get comfortable with that, they almost always bring it to your most mission critical use case because it's a probably is availability and durability, which is very, very hard to achieve any other way. And also, you usually need to move much faster. and develop and develop productivity is increased tremendously when you use this technology. But point is, you run the most mission critical thing and there is a resistance component which you need to scale and manage and most companies at some point realize that it's better to pay us to do that versus doing it themselves. Still, it's fully featured. Companies like DataDog, like Salesforce are still fully self-hosting, very large number of clusters and pretty successful with that. So it's not kind of crippled software. At the same time, a lot of companies like Netflix, they migrated from self-hosting to our cloud and are not looking back. One thing is about like how we approach that. Again, we only monetize the running backend cluster and we are consumption based. Practically you pay for what you are using. And it's very good model because for most companies, when we start with us, there is no big contract to sign. There is no like something. It's just, yeah, you can start using that and you pay as you go. And it's a model which works very well for them. And certainly I think it's pretty good from monetization point of view as well, because it's very much aligned to the customer value. And so far it works very well for us. One more thing is that people always ask, when are we going to change license? Are we going to change license from MIT? Because MIT is very, very permissive license. And you know, like promises don't... It's very hard to make promises because it's very hard to keep our promises with corporations, right? I can promise whatever, the more I can be out and something else happens. But I always tell people that... From business point of view, the worst thing Temporal can do is change their license. Why? Because imagine, we are still a relatively small startup, right? Like we are 250 people now. And when we started, we were like 30 people, right? When we started to monetize that. You go to the 500, like YCMP 500 company and say, okay, now you need to put your business, like you're planning to put your most critical use case on startup code. right for 30 people startup, and nobody will ever use you in production for that. So that's why having a fully efficient open source gives us people that trust. And that's why we guarantee because the moment we change the license, most of our customers just move from the platform because it will be 100% locked in. So we realized that it's pretty clear. So from a business point of view, staying with the MIT license and ensuring full compatibility between our proprietary cloud offering and the open source offering is just a requirement. And it's common sense. It's not just because we are good citizens or whatever. It just makes a lot of business sense as well. Yeah, we love open source in general, but again, it also makes a lot of business sense for us. Yeah, it's interesting. I think the thing that we've seen definitely in database companies as as they've gotten bigger, other companies like Amazon want to offer their cloud services as a part of the Amazon offering. And that's one of the real big risks is people just running your product in the same way that you are trying to operationalize it to make money. So have you all thought about like dual licensing or like is this just something that you're pushing off until it becomes an issue and then you'll sort of like reevaluate like what does that look like for you? You can guess that this question was asked by everyone since the even before we started the company. Obviously our first investors we've had long conversations about that. I think there are a couple points there. One is that I think Amazon in general became much friendlier to open source. We have very great relationship with Amazon right now. we have long term agreements with them. So it's one. Second is that and we are like transacting the marketplace all the time, right? Like a lot of people come from Amazon marketplace. Second thing is that we, the way we differentiate our cloud, besides just managing things, we actually built our own resistance. So Temporal is practically Temporal cluster system database, right? Open source supports MySQL, Postgres, Cassandra. And we build out a temporal internal database we call CDS, or Cloud Datastore. And this database is kind of written for a single API call. You can think of it as a database which is written practically to solve one specific use case. That's why we can do optimizations, which are not practically possible if you use any general purpose database, even the best one in the world, right? Like people say, why don't use whatever we can, you can. But it never would be able to match performance of tailored solution we are building for that. And that's why the AO cloud service, if you compare practically the cost of running OSI, open source, even not counting people, just against the AO charges, especially at large scale, you practically end up in a situation that is actually not more expensive and very frequently much cheaper. And then you start counting people to manage your infrastructure. It is much cheaper to run on the temporal cloud than self-hosting. So if Amazon starts doing that, unless they invest immense amount of time to build in their own kind of storage solution, which is tailored for that, then it would be very hard for them to match performance there. And also, if Amazon starts doing that, I think it will be the best thing which happens to us because the pie will get much larger, right? Because still, durable execution and temporal is still not the ubiquitous technology. If Amazon starts offering that, I can guarantee you it will become ubiquitous very, very fast. And I think we will find ways to differentiate. It is my... I'm not expert there, but my understanding a lot of companies which have suffered from that, they didn't have superior cloud offering to Amazon when Amazon started to offer that, right? We do have one. If Amazon starts hosting to portal open source, they wouldn't be able to compete with us on anything but like, okay, we will give it to... we will give it to people who have marketplace credits. Two steps. like people who do low scale and not enterprise like workloads. Also, don't forget to have simple workflow service. I was tech lead for that. And it's practically the same basic idea, just a little bit outdated. And I think Amazon practically logically deprecated that they don't promote it anymore. So for them, okay, there aren't Kinesis and Kafka clusters, so they still can do it. Yeah, that makes a lot of sense. It's just that's a that's a question that we like to like, ask a lot of folks who are creating open source businesses. So as we're wrapping up, we always like to ask future facing questions. So you've sort of like coined this term durable execution or definitely been involved in like developing the space. How do you see it evolving in the next like five to 10 years? What are the big unsolved opportunities in this space? Like if you have quite a few. One is just more runtimes, right? Like deterministic runtimes. And obviously WebAssembly is promising. We have competition doing WebAssembly based ones. The first thing I prototype actually when I started to pull this was WebAssembly. Unfortunately, it's just not practical because we want to have billions of parallel executions. Unless you're writing in Rust or C++, right? These containers are pretty heavy. So you cannot have a lot of those loaded. So it won't be practical for a lot of large scale scenarios. But the moment it becomes more mature technology, we will absolutely integrate that. Also, we can do runtimes like we do in V8 for JavaScript. So maybe even on the operating system level. But if you think about like, conceptually, this is the first technology which abstracts out distribution from developers, right? So because this process is not linked to a specific machine, so we practically do live migration that process seamlessly, you can think about that it can be kind of replace the process idea. of the operating system. So you can think about it as building an operating system for the world, right? Like this is like this real distributed operating system, which can have stateful processes. So I think it will evolve in more this kind of become more and more like operating system and have more services and more capabilities and more like operating system level features. How it exactly will end up? I don't know. Obviously, another big part is all this Argenic stuff. I believe that almost every agent long-term, if you care about state and it's kind of doing real work, will run on top of a durable execution as an orchestrator. I think this is the given. We have a lot of usage already in production by a lot of companies. I wouldn't be surprised that Temporal runs more Argenic workloads than a lot of... much more. AI known companies then because people who know Tim Portal just immediately start using it for those workloads. And just for example, recently, OpenAI publicly kind of say that, for example, the image use case, right, like when every time you generate image, we use an open chat GPT, it uses Tim Portal behind the scenes, right? The same thing, the cadets, they have this cadet server, right, like the coder, right, like this environment, it's all based on Temporal as well. So companies like OpenAI are using that, but you can imagine that more and more companies which do AI workloads will learn about Temporal. And we will do more stuff to integrate, like make it much more seamless experience for new devices as well. So yeah, I think this is... And then there is one more part which I think is very important. I mentioned this RPC, long running RPC. So we have this protocol called Nexus RPC, which is practically standardization on top of extension of HTTP, which allows you to run long running operations. So we want to extend MCP with that. So we want to extend MCP to support long running tool calls because we want a reliable long running tool calls. And we already have solution for that. And it works perfectly with Team4O. So we would be able to practically be a big part of playing with this ecosystem. So Temporal execution and Temporal will be a big part of tool calling ecosystem when you need reliability and cross-company calls. That sounds really awesome. Sweet. So that wraps it up for our questions this week. Thanks for coming on, Max. This was a really interesting deep dive into the architecture that powers Temporal. So thanks for coming on. Thanks a lot for having me.

This week we talk to Maxim Fateev, a co-founder of Temporal. Temporal started as a tool for Uber but quickly grew into a tool that makes distributed code exectution a breeze. Come hear what one of the poineers of Durable Exectution has to say!

https://temporal.io

Episode sponsored By WorkOS (https://workos.com)

Become a paid subscriber our patreon, spotify, or apple podcasts for the full episode.

https://www.patreon.com/devtoolsfm
https://podcasters.spotify.com/pod/show/devtoolsfm/subscribe
https://podcasts.apple.com/us/podcast/devtools-fm/id1566647758
https://www.youtube.com/@devtoolsfm/membership

00:00:00 Introduction
00:01:40 The Genesis of Temporal
00:05:10 Durable Execution
00:10:30 Ad
00:11:42 Temporal's Developer-First Approach
00:19:38 Temporal's Robust Architecture
00:21:52 Real-World Use Cases of Temporal
00:27:10 Handling Large Payloads in Temporal
00:30:12 Versioning Workflows in Temporal
00:36:14 Deterministic Code Execution
00:39:18 Temporal's Open Source Journey
00:48:13 Future of Durable Execution
00:52:00 Conclusion and Final Thoughts