Durable Endpoints: Build a production ready Deep Search in 30 mins

Just going to walk through Durable Endpoints, which is a really, really, really, really exciting thing that we've been working on and have talked about since actually last August. We are super, super stoked for this. And we're going to show you a bunch of examples. We'll have an example deep search thing that we'll run through, which honestly only took a little while to create. And we're going to talk through exactly what Durable Endpoints are and why we think they're so cool. and why so many other folks that we've shown in our access think they're so cool really can't wait uh yeah let's dive into it just in case you don't know me uh tony started the company a while back um previously i used to work in healthcare um we don't have the nhs in america so i was like oh we can we can improve american healthcare it was hard um takes a lot of work and it's very very workflow driven very event driven which is where we came up with the idea of ingest Essentially what would have taken us a day using Ingest would take months to build out manually using Kafka, events, queues, that sort of stuff. Ended up building the Ingest APIs based off of our experience working in healthcare. And the aim of Ingest is to allow you to build really, really complex products without worrying about infrastructure at all using an SDK, single line of code and you're good to go. So just quickly talking about the agenda. We've got a few things to run through. Essentially, what are driven endpoints? Why are we talking about them? And how did we get here? Where did the entire idea come from? And then we'll talk through how to build and reinforce APIs using driven endpoints, why it might be a good idea, and some really basic examples to show you the theory behind it. And then we'll talk about how they work under the hood, which is super fun. And I can imagine there'll be a bunch of questions, which will run through basically all the talk. There'll be some Q&A at the end. but I'm happy to answer questions as we go. And then we'll talk through an example use case using deep research. And then there's a little bit on roadmap of what we would like to do to add to durable endpoints in the future, which I'm super interested to get your feedback on. And then Q and A's. So diving into things to kick it off, what is durability and why does it matter? I guess like durability to begin with has been retries and reliability, job queues, VLMQ, that sort of stuff. And it's just essentially retries for background jobs. And that's cool and all, but it doesn't necessarily help with all of your application. If it's just about retries and it's just about single jobs or for example, workloads that run in the background, you're missing out on a lot. You're missing out on a lot of really good things like flow control. in case you wanna debounce or throttle or sort of like rate limit your workflows. If you're hitting an API, for example, and you're missing out on a bunch of things with observability. So I'll use undurability that you get, sure, all of the retries and the state management, but you also get flow control so that you can rate limit your users functions to maybe three a minute. Or in GitHub's example, if you're using GitHub logs as an API, like three a second for each of your tenants. and then you get observability, tracing, metrics, insights out of the box. And again, this is super interesting for building out your core logic. But that core logic has always been for background workflows. And those are async processes that run, you know, not necessarily in the browser, not in response to users' APIs. And workflows are great. We've been doing them for a long time, as have you. but it's half of the product. And the other half of the product are API endpoints that your users use routinely. And those API endpoints should be able to do workflow-like logic, including retries with durability without breaking. And durability in general is, you know, it's more than async retries, as we talked about the flow control, all that sort of stuff. And if you're building AI endpoints, which a lot of people are nowadays, you kind of want durability directly in your API endpoints as well. And ideally, you would have all of this without any extra infrastructure. Your API endpoints would work wherever they are using any framework on any platform that you're using for compute with all of the durability built in. And this is kind of important, you know, like if you're if you're doing agendic coding and you're copying and pasting errors from your API logs to solve things, or if you're using webhooks consuming from different services those webhooks are going to be posted to API endpoints that you host that should do things reliably and if you're doing things like product integrations you're going to want to make sure that your product integrations and your on your API endpoints for OAuth or for kind of lightweight product ETL work correctly on top of all of the AI stuff and nowadays AI stuff is just honestly a bunch of steps that run and sometimes your AI endpoint is going to fail I think. Anthropic is fantastic and sometimes Anthropics models just return errors because of capacity which is which is a tough problem to solve. So there is this gap right now in durability with workflows existing but APIs being sort of left. to the way that we've been working for the last 20 or 30 years. And in our view, we were thinking about this and we were thinking about how we can make this a ton better. And that's essentially where we came up with Durable Endpoints. Durable Endpoints are all of the benefits that you get from Durable workflows, from queues, but in your APIs, in your current framework, on your existing compute, without any changes or without any extra infrastructure. You get a bunch of stuff like automatic APM. So that's tracing observability metrics, which are really cool because if your API endpoints fail, then you get all of the errors, the traces, and the steps, which is really cool for a breakdown of what failed and why. And your AI agents can then take those errors and fix the endpoints pretty easily. You also get a bunch of durability, state, retry, flow control, essentially all of the workflow-like logic, including sleeps or wait for event, but within API endpoints, which is kind of sick. I'm going to talk through how it works. And in order to contextualize how it works, would really like to share this example code, which is an extremely lightweight version of Dribbble Endpoints using Bun. And it works on Bun, Dino, Node, works in Go, bringing its Python, just anywhere really. In this example, if you can see the code, you've got an ingest endpoint. And that ingest endpoint is a basic wrapper over an HTTP function. And this request is the standard library request object. The response is the standard library response object. And you can see that we're using a step.run. And then we're returning the result of a step.run into the variable foo. And we're using that in the response. And there's going to be no latency here. Steps are going to run. It's going to process as fast as possible. And then it's going to return the result to the user without any latency here. which is super cool. And if you do do things like step.sleep or step.waitforEvent, we're gonna handle this automatically in the following way. The user history API endpoint, not us. This can be a regular REST API endpoint. It can be something you give to your users. It can be your product. It doesn't matter. We're gonna run those steps and we're gonna checkpoint states in the background so that it doesn't impact your latency. We're not going to do any blocking operations. And on the happy path, your user is going to get the response without any overhead. We're going to flush the buffer to get that response to the client's browser as quickly as possible. And we just wrap that API endpoint and then save all of the state and all of the traces and all of the APM from your steps for free so that you can see that a user hit your APM point, there was a request, steps executed, and everything was successful. Then if something errors in your step and you want retries, or if you use step.sleep, step.waitforEvent to build a workflow, we are going to redirect the user to a run specific URL for that single request. And that URL is going to block and wait for the function to complete your original API endpoint. We are going to reenter that API endpoint, inject all of the states so far at the right time. So that's either after the sleep, when the wait for event has been received or times out, or after the time for the retry. We're going to finish that API endpoint. We're going to grab the result from it, store all of this in state, as you would expect from the async workflow style way of working, and then send that to the user that was waiting for the original API response. And that means that you get the reliability, the state, the management, essentially the entire workflow-like aspect. of durable workflows in your API endpoints with almost no trade-off, which is really, really, really cool. And we are really, really excited about it. The almost no trade-off is super interesting because essentially when you checkpoint in the background, there's no latency here. The only trade-off you're making is checkpointing in the background means that you're not checkpointing after every step. And so once things error, you have to block checkpoint all of that state and then redirect the user to make sure that everything is stored appropriately. So there's an increase in latency on the sad path, which you would expect because we're going to have to retry those steps anyway. So it's absolutely minor and really, really, really exciting. Any questions so far? We are going to go through a quick setup. I'm a NeoVim person. I haven't used OSWiTCH to cursor or any of the fancy stuff like Zed. We're gonna open up NeoVim and we're gonna work through the BUN example and show how it works, show you how to get set up from the absolute bare minimum example. And then we're gonna talk through it and then we're gonna go into the deep research section. So over here, you should be able to see this code. I'm gonna make this bigger. And I'm gonna quickly talk through how this works, right? Like you've got your ingest client, you've got your bun.serv with a port, which is the lightweight example because it's built into bun. And you've got some routes. And normally you'll have a request and a response. This is the simplest HTTP function you can build in bun ever. And in order to make this durable, we're going to wrap this with an ingest.endpoint. and that adds all of the state management the caching the retries the flow control to your endpoint um so it's about what you need to do then when you save this and run it let's run that we're going to go back to the browser and i'm going to show you the tabs we're going to hit this endpoint um there's no response because i didn't add one and we're going to add a response over here hello run this again uh refresh you'll see this so if we go over to the ingest server you're going to see two requests because we track and give you all of the metrics around what's happening and if you click on the run you'll see the output over here and you'll see some of the information the input the request stuff like that and in this example we've got two requests that didn't really do anything which is which is nice the first one there was no output and so on if we add a step it's pretty simple you can just do await step dot run Hello, step A, async function. And then we're going to return some data. And then we're going to add this data to the response. And this is your essential basic ingest steps, which you'll be familiar with if you've used this before. And then if we go back, run this function over here, you'll see hello, some data. The response is extremely quick. Go over here, hit refresh. Took no time at all. There's no additional latency. And then if we go to the Injust server, click on this, you'll see step A, you'll see the output, and you'll see the entire API response and the lifecycle of your APIs, which is cool, because then you get tracing for free, metrics for free, timing for free, real basic APM without doing any work. And that's pretty awesome for essentially understanding anything that happens in your API endpoints. We are going to add asleep to this, I wait step.sleep and then we're going to sleep for maybe five seconds to show you what happens when you go async and we'll add it after step.com. I should have done a watcher on this and if we go back to this endpoint and we hit refresh on this, clear this so that you can see how things work, bring this up. If we refresh on this, you can see that we've been redirected, the browser is loading. And then you can see that we return all of this information, including run complete with this data. And this proxy gives all of the raw information as it stands right now about the run. Realistically, in a real API endpoint, you would just return this body with this status. This is here to show you how we work. And I'm going to talk through this now. If we open up the server and go back to our latest function, you're going to see that we had Steppe run with some data. then we had the sleep execute and then we finished the function when we requested this particular endpoint the the the user hit this code and the user hit this step.sleep and we redirected the user using a 302 to this particular url and this url is over here it's at slash poll and there's a run id and a token and that token gives the user access to see this specific requests response and this particular url in code is what we've specified up here which is this redirect url we redirect the user to this particular url ingest then re-enters the function after the sleep injects all of the state directly into this api response we pick up where we left off which is how you can still see hello with some data. Ingest receives that information over here. And you can see that over here after the sleep. And we send that directly to the user, which is really, really cool because then you get a whole bunch of workflow-like stuff directly in your API endpoints, as we'd said. Yeah, I really love this. It's amazing. Super, super, super interesting. So this is the basic example. You can sort of see that you can run any steps. You can do step.waverevent, you can do step.sleep, you can do step parallelism. Everything that you can do in regular workflows using regular ingest, you can now do in your API endpoints with zero latency. This still took five seconds and it was extremely quick. 48 milliseconds end to end, including the HTTP requests. Yeah, it's really, really cool. And building from this, I will go into the deep research example, talk about how it works for something more complex. And I'll share the code, the UI, and walk through how all of this works using essentially two things, three things, ingests, driven endpoints, EXA as a web search, and Anthropic to tie everything together with the AI models. So let's quickly run through this. I just quickly want to talk about why we are doing deep research and how it's going to work. There is a planning phase, which will ask what kind of stuff we're researching. It's going to break all of the research topics down into multiple specific requests. It's going to fan out into thousands of topics and thousands of requests using AXA. um it's going to absorb all of that information runs specific solve agents to gather all of the information and contextualize stuff as you would expect from clock code validate the findings and then give you all of the responses back we think this is a classic example that pretty much many people do and it's important because it's a mixture of sync and async the sync stuff is the ai that kicks things off It's the chat experience. It's the back and forth between you and the deep research agent. And the Async staff is spawning background agents that go ahead, do all the autonomous work, do a bunch of searching and tying together the context as you've seen if you're using cloud code. So this is a pretty solid and good example that you've probably seen before. And so let's go back to NeoVim and get started with the code. Over here in a different team ops panel. I'm gonna quickly open this. I go over here from pmpdev. We're going to get started on port 3000 using a basic next app. I'm going to talk through how things work. And first, let's load the code. If we open this up, you can see that we've got deep research here. We've got a UI with some text box for what we'd like to research. And we're going to research something about queuing systems slash events because that's what we do. And so we've got this layout. And we've got a few API endpoints that are going to kick things off. These API endpoints are slash research. And then we have a route to kick things off and initialize the research. We have an API endpoint for clarifying the request, which is going to take the initial request and then break it down into specific sub requests, ask you for topics so that we can make a better prompt for for the deep research agent and then we have another API endpoint for events for fetching various states because this is also if you reboot the server if you reboot ingest we keep a bunch of state around locally so that you can come back to any previous research and that is essentially a hacky database for this particular demo so let's go for a quick example first research events and multi-tenant queues. And then let me open up the network panel. You can see that this is running. I'm actually gonna go back and do that again. Events, let's clear this and start the research. You can see this is running in the clarify endpoint and the clarify endpoint returns some results. The results are just regular API response. And that clarify endpoint is actually a durable endpoint. We go over here, here clarify. You can see that we're calling ingest.endpoint and we're wrapping the next request and we're using steps. So if there's no topic, we're gonna return an error and we're gonna use steps to generate clarification questions. And if we go down here, this is literally just a quick AI call. to Anthropic which is going to generate this particular result, this object and then it's going to return the questions that we'd like to ask and then after it runs that step it's going to return a response. So if we go back to our UI, these are the questions that Anthropic would like to ask. What are we interested in? Architecture, implementation, strategies, performance optimization and such. I'm going to move this over to the right so it's easier to see. And you can see this particular preview here. And if we go over to ingest, you can see this question pop up here. Research clarify. We generated the clarification questions. You can see the output from the particular step, which means you get full observability into all of your API endpoints, how AI works within your API itself, any of the tools, any of the steps that you run, everything kind of shows up, which is really cool. and really really good for observability and debugging and local testing especially good for iteration if you're iterating on AI once we've gotten that we're gonna we're gonna ask for architecture actions implementation strategies maybe we're gonna go for distributed systems alongside message brokers And we want maybe research papers and implementation details since we're going to change this to 2021 plus over the last four or five years and then hit start research, which is going to hit this research endpoint, API research. And you can see the durable endpoint. So over on, you can see the information appear. We'll talk through how this works. We'll talk through how to build it. And we'll talk through how this works with the durable endpoints in a second. We're going to quickly go back to the ingest server. You can see this particular durable endpoint running. You can see that it's generating a bunch of queries. You can see all of the searches to EXA. You can see the results that are coming through from this research using EXA as an API to make this incredibly simple. And it populates in real time, even if this is a long running durable endpoint with a bunch of parallel requests, which is super, super cool. uh diving into the code to begin with we kick things off with this particular this particular endpoint here which was the clarify endpoint this clarification endpoint is an ingest endpoint that takes this request and it has a topic which we then send through anthropic essentially this is a standard api endpoint that we've added a step dot run to so that we can observe the ai call and if the ai fails we can retry automatically and the user won't notice that anything's wrong because the 302 redirect will send them to a new url cause is handled so there's going to be no errors it's going to wait for this particular response and everything will just work which is really really cool in case your tokens are limited in case model providers are down everything will will be seamless all we've all we've literally added is a step to run and this generate clarification questions AI call is an anthropic AI call using the generate object from the AI SDK. So we have just added a zod schema to ensure that the API response fulfills what we'd expect. And you can see that if we remove this step.run we would get none of the insight of this particular AI within within the dev server, which means you wouldn't be able to understand anything that's happening. You wouldn't get the failures, the retries and so on. So yeah, this is pretty cool. Go back to the code over here. Once clarified, we hit the research endpoint and this research endpoint is the main endpoint up here. This research endpoint walks through a ton and we're going to talk through the core part of how this works. Again, wrap with an ingest endpoint. We pull out all of the URL parameters. And this has a failure rate. I'll show you a demo of how this works on failure, but it does exactly what I showed you within the BUN example of handling retries and so on. And if there's no topic, no identifier so that you can treat this as essentially a database, we're going to return an error. And then we emit some progress, which is essentially saving progress within your database. and we have a step dot run. And this step dot run gives you all of the observability that you see here from these searches. And you can see that we're generating the search query. And generating these search queries, again, is a really small AI call. You're a research agent. It gives it the current date, gives it the clarifications. And then it asks the user to... it asks AI to generate a bunch of information for us. This is all cached, memoirs over here. I'll show you this information down here, generate queries. It returns all of these particular queries that we're going to pass through to EXA. Go back to this code, you can see this right here. We're going to save some of this to the database as well so that you can show progress to your users. And then we are going to call this deep research function which itself has many, many nested steps. And the deep research is going to take each of those queries, these questions that the AI has generated, and it's going to promise.all with step.run search EXA for each of these questions. And it's going to pull all of the questions through to EXA, which is going to return a bunch of sources, which you can see here, search results. And these search results are then going to be filtered. stored in the database via emit progress. And then we are going to push them into a results array, which we use to then extract learnings from each of those particular sources. So you can see here that generate query, for example, the query is multi-tenant event queue architecture patterns, distributed systems, 2024, 2025, 2026. And down here in search, One of these is passed through to Exa as a separate step within your API endpoint itself. If these fail because your tokens are consumed or because your rate limit has been hit, it's going to retry. User won't even know, even if it's a regular API endpoint that just does a fan out to hit many, many, many API endpoints. And this is going to return a bunch of papers. It's going to return a bunch of websites and so on. Essentially, we take all of that information back. It goes into the learn aspect, which is then going to take all of the context from all of the exit things, condense it down into a bunch of learnings that we then show to the user over here. Let's find out where that is. So once it's finished, returns all of this information down here, which is pretty cool. Essentially two API endpoints and you're done, even if you have issues, failures, everything kind of restarts. So that's the TODR. We're going to recurse with follow-up queries. So it's going to do a multiple depth search over each of the learnings multiple times and just call itself over and over again. So this API endpoint runs for a good couple minutes. We can find the actual timing within Ingest's DevServer over here. This one ran for four minutes, nine seconds with many, many steps. and the last one generates a report with all of the information from its recursive learning taking the previous examples in the previous context going through generating more questions to figure out what happened again if we take a look at the code the API endpoint is pretty basic this is the deep research function and if we go back to the actual research function up here in route.ts it's a single get with ingest.inpoint. Steps are automatically stored in async local storage, so you can use them within nested functions and so on. And then we hit some anthropic API endpoints using step.run for observability, for traces, for retries, for flow control. And then we do the same thing within the research function, just calling step.run for every AI call that you're making. Everything works seamlessly. This is essentially how to build deep research using EXA. Real basic. AI call to generate queries based off your topic. EXA to search the web, search archive, all these sorts of resources. Pull back that information, pass this context into another AI agent. Do this in a loop, depending on how many times you want to fan out, and then recurse through each of those trees, and you're good to go. It all works in one simple API endpoint that runs for four minutes and you don't need to think about state or durability. On that note of state and durability, if we just go back to why we've done this, using steps means bypassing things like job queues or workflows altogether if you build them in your API endpoint. You don't need to worry about workers, you don't need to worry about state persistence and all the associated things you would normally have to worry about when you're building AI products. Also similarly in case things fail, for example if you're using a third party API and it's not AI, you don't need to worry about any of this stuff either. So it gives you a bunch of this stuff free, which is cool. And you just quickly want to talk about what's coming up in the roadmap for Dribbble Endpoints. improve streaming. Right now you can see all of this information appear in the UI. We want to improve a bunch of how streaming works for sync end, async endpoints overall. And we want to add 100% flow control compatibility, including what's upcoming in our OG roadmap for workflows, function level concurrency, and so on. And while we give you a bunch of this stuff, and we give you a bunch of APM, For example, we give you the traces, the metrics, the timing, the duration of your API endpoints. It'd be really cool if we could put this all in dashboards and show you information, insights, alerts on how your API endpoint fails. So the roadmap is pretty stacked. We want this to be something that you depend on. We want this to be something that you can add to basically every API endpoint that you have so that you can get really, really solid information, build things really quickly without worrying about infrastructure. And then quickly, I know that we've only got 10 minutes left. Q and A, the code is all open source. It genuinely took us almost no time to build because it's now essentially as every product is using this a few API calls. You can go on ingest deep research. So on GitHub, you can see everything. You can test this out locally, completely anonymously because of the dev server. You don't need an account. Just muck around with it, see how it works. It's entirely open source as well. The way that it works, the redirects, the logic, the state, the checkpointing, everything is entirely open source. And of course, it runs on your own machine. So you have full control of how everything works and the security. There's one thing we didn't touch upon, which is security. Ingest has an end-to-end encryption middleware. And that is really cool. After working in healthcare, you know, we don't want to see PHI. That's pretty cool with API endpoints as well, because if you're handling sensitive data in API endpoints, you can still enable end-to-end encryption. The step and the API response is going to be encrypted. Even when we re-enter that API endpoint, that data is going to be encrypted when it's sent to us. We don't see anything. And then when we... redirect the user to your proxy to bypass cause we redirect and we inject the encrypted state into your api endpoint which then decrypts it using the decrypted key and then sends it to the user so we don't see anything um if you if you're worried about security as well which is uh which is super cool um in terms of questions a quick comparison with a regular ingest call from an endpoint main difference is that you can't return stuff, one sent to ingest and the regular durable functions. Yeah, totally. I think like if, I kind of alluded to this in a previous slide. And let me open up that one so that you can see things. Regular ingest functions are cool. And they're cool because they give you a bunch of workflows, they give you all the steps, they're event driven. And that means that if you have one event, it might run 10 functions. You don't need to write. the invocation of 10 RPCs inside that API endpoint. You send an event, everything works. But it's obviously in the background. And you can totally hack around that background workflow using things like real-time to make it appear as if some of those workflows run in the foreground as a synchronous API endpoint. But while we like that, If you want things to be synchronous, if you want your users to be able to invoke a workflow themselves and have steps, you can use durable endpoints inside a regular API endpoint that you host. And it's essentially like building APIs, which we've done for a long time. And you can call those API endpoints using fetch. or using Axios back in the day. Axiom, was it Axiom? Prior to fetch being in the ES5 standards, it doesn't matter because it's all just standard library stuff. You fetch your regular API endpoint that you serve inside your API. It responds to the users and everything just works as you'd expect. There's no new programming model. There's no workflows. There's no real time. There's no, yeah, Axios, that was the one. There's no new programming model. There's no hackery that you need to do with broadcasting to the front end. It's a regular API that does exactly what you need. There's no latency impact because we also flush the buffer to the browser and checkpoint in the background. And so they're for different use cases, it turns out. Durable endpoints is for running APIs reliably. Durable workflows and the classic ingest functions are. for running background workflows, async stuff, fan out, events, the background processes that most people have as well. Yeah. TLDR is a comparison. Yeah, they both do workflows. They both do workflows. API endpoints and durable functions are super low latency and your users can invoke them because they're API endpoints. Background workflows, the async functions from ingest are event driven. They have a bunch of flow control. which is interesting, like batching and so on. Batching won't be coming to dribble endpoints because batching is a really interesting feature for background workflows specifically. And it's slightly higher latency because it's asynchronous. There's an event that will call those background workflows and then those workflows are triggered. How are we showing the status updates? Yeah, so showing the status updates, there was an events API. Let me see if I can show you that. That events API is here and it gives you a bunch of information around what's going on with the deep research to show that status update. We do have a bunch of work for streaming from Dribble endpoint so you can return a readable stream. The thing that we need to do, and that's on the roadmap and we're doing that next week, is allow you to return a readable stream after the function has gone async. So for example, if you've got step, step, sleep, and then you return a readable stream the user is going to be redirected to another url we're going to make that essentially work using readable stream so you get the exact ssc output from your api calls like ai and everything will be super transparent your user won't even know it'll be the same kind of seamless experience so ssc support is is literally underway um would you ever combine the two dribble functions your dribble endpoints like totally yes like 100 yeah We do this. We have some durable endpoints and we also have durable functions. For example, if you've used our cloud and you've used, for example, bulk replay, bulk replay is actually a durable workflow. It's an async function that will make a SQL query to your ClickHouse data because we give you ClickHouse and all of your runs and events are stored in ClickHouse. We read the failed runs. that you've selected for replay in a step and then we iterate through those and we do bulk events to create new function runs um that's a durable function that exists in the background you can imagine that there's an api endpoint that should invoke that particular async workflow which can also be a durable endpoint and so you can do you can do invokes within durable endpoints you can send events to run background functions in durable endpoints Essentially on our behalf, because of the benefits of endpoints that give you automatic tracing, metrics, observability, we kind of favor using durable endpoints for basically everything so that we get much more observability plus a little reliability out of the box and then durable functions for the stuff that runs in the background still. Because durable functions use gRPC and they're long running for the stuff that doesn't need to be an API endpoint, we still use those everywhere. Any other questions? We've shown this to a bunch of people and it's really cool. It's really exciting. We're really, really happy and excited about where this goes. There's going to be a bunch more. Yeah, it's stable enough. Yeah, yeah, stable. We use it. It uses checkpointing, which we released for ingest functions a while back at the beginning of the year. Checkpointing is in v4 of the SDK. and it's going to be the default mode of working in v4 of the SDK. And the preview of the v4 SDK was released today. It's very cool. Middleware is so much nicer. The typing is so much nicer. We have support for the beta of TypeScript 6 in v4 of the SDK. So checkpointing is like, there's no downside to checkpointing. Enable everywhere. It uses all the stuff that we built for checkpointing, which is 100% stable. and good to go and yeah it's essentially an extension of the checkpointing work that we did so stable no downside and the other other part to say your API endpoints will never stop working if checkpointing fails for example which it won't but hypothetically speaking let's talk about like disaster scenarios if it fails your API endpoints are going to do exactly what they do now so in the worst case you have an API endpoint that exists exactly as it does today in a disaster scenario where AWS GCP, because we run on multiple clouds, both are down and things are completely fucked. Sorry, it's language. Completely broken. That won't happen. And so, yeah, it's stable. There's no downside to it. Sorry about the language. Turns out that employees in the company might be having bets on whether or not I swear in every talk that I give and I apologize. I got all the way to the end. Okay, unless there are any other questions, I appreciate your joining, appreciate your looking. We're really, really excited about this. We just want it to be really easy for you to build products, for you to build what you need to build. and to make it really effective. So yes, ORPC support for sure. We need to do more ORPC support. Yeah, thank you for, thanks for the time. You know, I think time is super valuable. So I appreciate your being here. And if you have any questions, feel free to ping us on Discord. Feel free to send me an email, tony at ingest.com. Always around to help the team and love learning what you all have built. helping you build things so yeah thank you so much take care folks enjoy your weekend and if it's cold where you're at stay warm bye

Deep research was the breakout AI feature of 2025. OpenAI, Gemini, and Perplexity all shipped it.

​But under the hood? It’s 40+ API calls across recursive search and synthesis—any of which can fail while your user waits.

​The usual solution: Queues, workers, state machines, retry logic... all way too much effort if you need to move fast.

​In this workshop, Inngest CEO Tony Holdstock-Brown will share a different approach. He'll use our new Durable Endpoints feature to build a full deep research system with automatic failure recovery. No queues, no workers, no infra to manage.

​He'll cover:

​- Why durability is moving from background jobs into API endpoints.
- How to wrap recursive AI workflows in durable, auto-retrying steps.
- Live failure injection—without losing work or re-running expensive LLM calls
- Real-time progress updates for long-running research

​Hosted by: Tony Holdstock-Brown, Co-founder & CEO at Inngest

​https://x.com/itstonyhb
​https://www.linkedin.com/in/tonyhb/

Chapters:
00:00 Intro & Overview of Durable Endpoints
01:27 Background and Inspiration
03:24 What is Durability & Why It Matters
06:02 The Gap: APIs vs Workflows
07:59 Introducing Durable Endpoints
10:10 How Durable Endpoints Work
13:11 Live Coding Demo: Basic Setup
17:17 Adding Steps, Sleep, and Observability
20:05 Deep Research Example: Setup & Context
24:01 Deep Research Example: API Walkthrough
29:16 Deep Research Example: Code & Execution
32:01 Durable Endpoints for AI & Complex Workflows
34:08 Roadmap & What's Next
36:46 Q&A and Open Source Details
40:43 Security, Encryption & Final Thoughts
43:38 Comparing Durable Endpoints vs Durable Functions
44:49 Thank You & Closing
47:36 Combining Durable Endpoints & Background Workflows
52:00 Wrap-up, Stability & SDK Updates