Stateful Next.js Serverless Functions - Pre Next.js Conf 2024 meetup

My name is Tony Hortzapron. I work at Ingest, one of the founders, and I'm here to talk about stateful Next.js serverless functions. And that kind of seems like an oxymoron. Stateful serverless functions isn't necessarily a thing. You might question, why does it even matter? That's a really, really good question. I think one of the interesting things is that the bar for product has gotten really high. As I think it's, was it pronounced foul? As you showed, you can build UIs really, really quickly. And Next.js has made it really, really easy to iterate alongside Bercel with serverless functions. You've got your branching previews, stuff like that. It's really, really hard to build modern products. And it turns out that in backend, it just hasn't really kept up. It would be ideal if you could deploy something like Next.js with your entire serverless functions, your backend, end-to-end, not just your CRUD stuff and have everything work with branch deploys, stuff like Neon, Tiedin. And I'll quickly talk about some of the problems and some examples. Right now, it seems as though serverless functions might only be good for CRUD. Not in that feedback is bad and, you know, posting to a database is bad, but... if you want to build complex things like marks's video pipelines which you shouldn't do you should just use marks you can't really do that in a serverless function and if you wanted to rebuild all of your indexes from strappy say for example for rag or you wanted to do embeddings that sort of stuff is really tough to do in a serverless function because you've got timeouts it kind of errors you might have memory issues it's in general kind of hard or if you take something like recent which under the hood is like okay cool you you send emails, but you need to do a lot of complex stuff with timing. The main verification, it turns out, is a little bit of a challenge. You can't do that on a serverless function because DNS is slow. The user might not have updated their DNS settings when they begin clicking the button. It might not propagate it, so you need to do this sort of annoying reschedule flow. You need to add state to your serverless functions in order to make it work. And this is where the sort of Next.js split happens when you deploy your functions to Next.js. You've got your server routes, you've got your used server, all the fancy stuff that came out with the next 13, 14, 15, which is cool. But then you've got your backend somewhere else, which kind of sucks. And then what do you do? You pay extra money to try and sync up your branch deploys, or you try and figure out how to get branch previews working with your backend. And then you try and figure out how to make that work with queuing systems. Doesn't quite work. The ideal state would be if we can just push all of the state to serverless functions. and you could make your serverless functions work end to end so you could have one mono repo build all of your code ship and then kind of call it a day and the idea is what could we do to make that happen how is this possible some of the complex flows we talked about here simple steps you can walk through resends dns propagation and flows as a series of three simple steps And these steps can happen on serverless functions if you inject state, which we'll talk about in a second. Firstly, you know, ping the DNS servers, respond, see if that works. If it didn't work, reschedule the serverless function to run at some other time. You could be smart about and take the TTLs from the DNS server. Maybe they didn't update that yet. So there's no TTLs. You do it in like 60 seconds or two minutes. It doesn't necessarily matter, but you just reschedule the function to happen. And then once you ping the DNS records, rerun it again, update your internal records. If that failed, which it wouldn't if you're using Neon, because they're great. If that fails, great. Reschedule the function, retry, and then finally send your notifications to the user to enable the account because, you know, spam is a thing and you don't want that to happen before the DNS records are verified. Relatively simple flow, three steps, not hard to do. Each one of them can fail as we've talked about. And you also have the same thing with another company who does, you know, AI. In this case, they are taking content from Strapi. analyzing your entire website, figuring out what pages you have, what content they look at, figuring out the keywords that they target, and then writing content to talk about those keywords for SEO. And all of those are kind of crazy multi-step pipelines that happen in the background that would be really hard to do on serverless functions unless you had some sort of stateful orchestrator. And that makes everything particularly easy. It turns out that it's only as possible. It's not actually that hard to do. And the answer is workflows, which are essentially step functions. This is an example of said workflow. If you ignore some of the stuff over here with the setup, you can see that there's step.run to get a page content, and then the step.index page to save to Pinecone or Neon, because pgfactor is great and you shouldn't use two databases. In this case, if the first step fails, we can reschedule the step function run the step again because you have state injected into the function once this finishes we save that step run data in some function run state rerun the function and inject that data directly into the function on any platform serverless so the cell or it could be it could be anywhere and then we skip that step run the next step and this allows you to build really composable workflows If you do step.sleep, for example, if you want to do the DNS propagation thing, you can say, Hey, reschedule this function to run in a minute. And then you can start building your entire backend in a really simple composable way without having to leave Next.js without having to deploy separate services, which is, which is kind of cool because if you can do that, you get all of the benefits of Next.js and the environments that you get with the sort of modern toolkits like the cell. And if you're building these really complex things that do a bunch of AI as steps, which a lot of people do. you can then start testing everything end-to-end with branch deploys and branch previews and it's kind of super easy you don't actually have to change anything the definition of a workflow pretty simple you have function id that's triggered by an event and uh super simple to run in general kind of like memorization on steroids really i don't know if uh if you if you looked into like memorization and caching but realistically cache a step dot run inject that state into the function and you're you're kind of good to go um this is super easy step.run is a simple api that i think a lot of people will start using we obviously offer it at ingest but i think a lot of people are going to start offering this api in the next couple years which is super interesting because hopefully this can become a standard such that it's really easy for you to build simple orchestration without having to worry about queuing systems ever in general and if you can do that then Largely, you've got front end, which is super easy. The database, which is kind of handled for you. And then everything else that you typically do on the backend can be rolled up into one platform, which makes development a lot easier. So it gives you a bunch out the box. Way better system to develop with than having this separate backend. And kind of cool. Yeah. Any questions? Super quick. Just from your perspective, what are you seeing? Make you see how this kind of pattern might be adopted through? almost like open AI's SDK. Yeah, yeah, yeah, totally, totally. So we had, the question is, why will this turn into a standard? And what patterns or signals are you seeing that will make this a standard? In order to answer the question, I'll talk just historically real quick, why we built it, how we built it, how we designed it, and then what we've seen over the last, like, just two years. Built it two years ago because it was totally necessary to make a bunch of healthcare flows work. Healthcare is crazy. do not join a healthcare company if you are in a healthcare company full respect to you very good um we released two years ago and since then we've seen a bunch of people take step.run and actually run with it we've seen a bunch of other people from like yc take it and run with it which is cool we've also seen a bunch of other companies like netlify try and introduce step.run which they did at their conference like last week and it is literally our api which is cool cloudflare have also signaled that they want to build the same thing obviously it will be on cloudflare's system And it will be using the same API. And so in general, I think the API that we've built will end up becoming a standard that will be on many different clouds, which is super cool. And in general, that should mean, you know, these sorts of systems are the way that we build. Don't forget the word. Would you like to compare this? Messaging systems are cool. They're good. And they've existed for a bunch of time, but they're sort of commoditized, you know, they're dumb pipes as it were. If you can build functions that respond to events, number one, yeah, send your events over to ingest or some other platform. Number two, yeah, you get fan out. So many serverless functions can run, but if you can build it in a smarter way, such that if we take a look at some of these things, the function knows the events and it knows the event name. We can take the types, the schemas, just like you get with Ava and whatnot, and build a schema library. You can also store these events through flow control. So you can say, you've got a high volume function, sales kind of expensive. Maybe you want to batch a thousand events into one function call so that you lower your costs. That's super easy and it's just a batching config in your function. So largely, same decoupling, better developer experience. You can test locally with one command and it runs anywhere, which is... One of the tough parts with serverless, like this is kind of why I feel like, I mean, I've loved serverless for some time, but it's been hard to develop end to end because it's, you know, crud-like you can't build these complex backends. And so similarities for sure. Very, very, very different use cases. And actually there's a lot layered on top, like batching, debouncing, throttling, concurrency to make it work. So yeah. Yeah. So these are. yeah you send an event over from any particular system you can like allow or deny list ips or event names for public keys if you want to send send them from the front end for example page events or if you don't want to do that because security is a thing um yeah server actions um can you subscribe yeah theoretically um we actually don't do real time right now because well largely we see a lot of people building really complex flows the flows that i've described over here are all largely back-end heavy um if you were going to build an api endpoint like yeah you might want to subscribe to that and that's actually something that we are building now um right now it's largely focused on your back-end orchestration um yeah i mean it's it's totally easy and possible to build um you just want to make sure that like things are low latency things are fast and uh security is a is a thing like if you're doing back-end orchestration typically kind of keys to the kingdom you know you might be we've got people that do insurance for healthcare and they run a bunch of insurance flows and you would not want people to be able to get the output of those results and so yeah just making sure that everything is end-to-end done really nicely is important before rolling out so maybe you don't mind going back to the snippet i'm curious how close this is to you know uber cadence or other or for orchestrations and like for example this pgurl Are you doing things based on where you serialize the closure as you're having to have this file be specially imported by ingest? Or are you doing checkpointing at each step? Yeah, yeah, that's a good question. So many considerations here. And I'll talk about some of the considerations, and I'll talk about Uber Cadence and how that stuff works. Your considerations with serverless functions are obviously time limits, memory, shutdown, random failures. Your considerations with functions, they are checkpointing and how to make it as close to exactly once as possible. For people that don't know, Uber Cadence is a system that came out something similar like this, like 2015, 2017. APIs are, you can look at them and see for yourself, you know? But they built a lot of similar things and principles and respect for them for that. Somewhat similar, event-driven, which is a big difference. One event can trigger one or more functions instead of the RPC star. Because it's event-driven, you can do interesting things like, step dot wait for event with an expression and then you can pause your function whether it's on serverless or you know kubernetes or whatever wait for something specific to happen with a timeout and then resume the function automatically when that then is received or if it times out you know the variable will be not and you know that that thing didn't happen um because it's event driven you can do things like look back so if you've got click house or like that sort of stuff you know no race conditions basically um very similar yeah we do check pointing after each particular step because serverless functions and timeouts are a thing and the way to sort of manage that is checkpoint after each particular step, rerun the function and make sure that it starts anew. Although there's different modes of running such that you know you can just automatically run down which reduces latency but obviously in serverless functions may not be what you want. So you memoize the steps and don't rely on serializing the code here? No we memoize the steps and this is something interesting as well like If you're going to build these workflows, which I think over the next five years will become super common, you're going to end up changing them over time. And I don't know how many people here have worked with queuing systems or built some complex stuff on the back end. And they realize that, fuck, we need to change this code. And this code has already been enqueued like a million times. And like, how are we actually going to make this work? That is no fun. Versioning in this sort of thing is kind of weird. You can see that we add step.run and there's an ID, which is get page content. Each particular ID is hashed with an index, which is the number of times you've seen the ID. And that becomes the hash key for the function state. So if you want to change things, it's super easy to change the ID and it will rerun and it will rerun that step. Also means you can do steps in a loop without doing your own identifiers or counts because, you know, we store the counts for you. And then it means if you reorder things, technically you can do one of two things. You can store the entire stack, which is the order in which you saw every hash. Or you can just do a straight lookup and say, you know what, you reordered it. You're in like lax mode. I don't know if people remember you strict back in the day with JavaScript. Like, why don't we just throw that in here? And then you get your own sort of stuff, you know? This format and this API is, I guess, like one of the principles that you learn in engineering is like explicit versus implicit. This is extremely explicit to the point where I think that's why people are taking this and running with it overall. And it's much simpler to reason about. Yeah, yeah, yeah. The question is, why wouldn't this timeout? And are you rerunning this? So yeah, in between each step.run, function state is memoized, cached, re-injected into the function and re-ranned. And that means that each particular step gets its own timeouts by default. And that is easy. Obviously, if you've got a 900 second timeout on the cell, you still have to abide by those laws. The same with, like, Lambda, you know, you've got your 15-minute stuff. That is... sort of a definition, a defining feature of serverless functions overall. Google Cloud Run, I think, is one or two hours, which is kind of crazy. They're actually pretty cool. If you want to do serverless functions, that is kind of, you know, this runs on your platform. So you get your own platform restrictions. We just rerun the function on every step. Like, I just can't exactly how that works, but I... Yeah, yeah, yeah. Orchestration, you know, we call your function when it needs to be called. And after a step.run runs, it sends us back the data. function terminates due to magic that we've got in our SDK and then it schedules another function call with the updated state to hit that function it sees the updated state skips the previous steps and hops straight to where it needs to be I get this because it's not wise so is this serverless function timing out just like a fact of life? I mean is it? yeah I mean it would be to a degree. Nice if you did that. But then I guess your questions are like ECS is a thing. It's probably more cost effective to pay for ECS than it is to pay for a bunch of functions running. Functions do time up. Platform specifics. We take the stance of we don't ever want to run your code because security is a thing. Come from a healthcare background. I don't want you to poke holes in your VPC to give us database credentials or that sort of stuff. You figure out where you want it to run whether it's like bare metal or whatnot. I know it will run serverless function timeouts though. Yeah, I don't know why the cell has such a small timeout that frustrates me Yeah, yeah, yeah wait 14 days wait for event wait for event is cool because then you don't need to use like any crazy signals You don't need to report any state you can do automatic cancellation by configuring cancel on with an event as well So if you're building some sort of AI flow and the user hits cancel send an event that says it was cancelled your functions will automatically terminate um yeah similar apis um just ergonomic because it should be easy for people to use in general cool

Tony, co-founder, and CEO of Inngest, walks us through the myth of Next.js Serverless functions, showcasing how Inngest helps transform them into Durable Functions, enabling companies such as Resend to build complete SaaS with Next.js.

#nextjs #vercel #serverless #workflows