Durable Execution: How AI Agents Survive Crashes

Welcome to AI Techbook. Today, durable execution, what it actually takes for an agent to survive a crash without redoing its real-world work, and the one test that gets you there. The first time one of my agents crashed mid-run, I did the obvious thing. I restarted it. And it charged the customer a second time. That's when I learned the thing nobody tells you. My agent had been running for an hour, doing real things, Charging a card, sending an email, writing a row, the crash didn't lose my work, the restart re did it. I'd always thought surviving a crash was checkpoint plus retry, save your place resume. But that framing hides the hard part and agent steps are side effects out in the world and a naive resume runs them again. Durable execution isn't save and replay, it's making replay safe. It comes down to one question you have to ask of every single step. Can it be safely replayed? Let's start with why replay is so dangerous in the first place. Look at what your steps actually do. Here's why replay is dangerous. In a pure function, rerunning is free. Same input, same output, no harm done. But your agent isn't pure. Every step it takes does something out in the world that rerunning can't undo. Charge the card again. and the customer pays twice. Send the email again, and they get it twice. Write the row again, and now you've got to duplicate. The action already happened the moment the step ran, and the world doesn't have an undo button. The problem was never saving my place. The problem is that rerunning a step re-does its effect. Durability has to answer a harder question than where was I? It has to answer, is it safe to run this step again? Let me make that danger concrete with the exact bug that fooled me. Here's the bug and it's a sneaky one. A workflow checks a promo end date against the current time call, then decides whether to apply a discount, then charges. First run, it's before the promo ends, so it applies the discount and charges once. Then the process crashes and the engine replays it from the top. But on replay, the current time call returns a new value. Now it's after the promo ended. The branch flips. The replayed run takes a different path through the code than the original did. On that path, the charge fires a second time. Nothing crashed wrong. The retry worked perfectly. The replay didn't reproduce the original run, because a live value sent it down a different branch. That's the failure checkpoint and retry literally cannot see. How do durable engines stop this? With a simple idea, they don't rerun your finished steps at all. Here's the core mechanism in temporal, restate, and dbOS. I'll share it. There's a journal and append-only log. Every time a step completes, the engine records its result in that journal. Now, when the process crashes and the function reruns from the top, something different happens. For every step that already has a journal entry, the engine does not run it again. It replays the recorded result instead. So the head rewinds to the top. Every finished step lights up and replays its stored result instantly, no re-execution, and the function lands on the first step that never finished. Only that one actually runs. That's the whole trick. Already completed steps are not re-executed. Their recorded results are replayed. The charge that already happened replays done. It never goes out again. But this only works if the code around the journal obeys one strict rule. Why does the orchestration have to be perfectly deterministic? Replay only reproduces the run if the code takes the exact same path every time. That's the rule. The orchestration code must be deterministic. Deterministic means, given the same recorded history, the code runs the same way and makes the same decisions. Anything that isn't deterministic cannot live in the orchestration path, the current time, a random number, an outside call to a service, a database read, a model call, It gets pushed out into a step recorded in the journal. Watch the double charge bug get fixed. The current time call lifts out of the orchestration into a recorded step. Now the branch doesn't depend on a live value. It depends on a recorded one. Every replay walks the identical path. Determinism stops a completed step from rerunning down a different route. But there's one step it still can't protect, the one that got interrupted. What happens to a step that fired its action? then crashed before it could be recorded. The journal protects every completed step. But picture the step that sends the charge, and then a split second before it writes done to the journal, the process crashes. From the journal's point of view, that step never finished. There's no recorded result. On replay, the engine runs it again, and the charge goes out a second time. This is unavoidable. A step's action and its journal record can't be made perfectly atomic, so every durable engine gives you at least once. Not exactly once. It will run your step at least one time. Sometimes after a crash at exactly the wrong moment more than once. The engine cannot promise exactly once on its own. That throws the responsibility back to the step itself. The side effect has to be safe to run more than once. How do you make charge the card safe to call twice? It comes down to one small field. You attach an idempotency key, a unique add for that one specific operation. Here's how it saves you. Two identical charge requests go to the payment service, each carrying the same key. The first one arrives, the service charges the card and remembers that key. The retry arrives with the same key, the service recognizes it, and instead of charging again, it returns the original result. Same key, same operation, one charge. The duplicate hits a seamless key gate and gets deduplicated. The customer is charged exactly once. That's the missing half. The engine gives you at least once delivery. The idempotency key makes the effect exactly once at the service. Replay safety needs both deterministic replay and idempotence side effects. Now every step is safe. Finished ones replay from the journal and the interrupted one retries against a service that won't double apply it. The whole thing reduces to one question. Here's the whole discipline on one screen. For every step your agent takes, ask, can it be safely replayed? There are two halves. On the left, keep the orchestration deterministic. No live time, no randomness, no direct outside calls in the decision path. Push them into recorded steps. On the right, make every side effect idempotent. An idempotency key, or a check then act that's safe to repeat. When both are true, the step is safely replayable. Here's the rule I'd tattoo on the wall. If rerunning a step would do its real-world action twice, it isn't durable yet. Notice durability isn't one feature you switch on. It's a property you have to be able to claim about every individual step. Get both halves, and a crash becomes a non-event. The agent replays its finished work and safely retries the one step that was in flight. What do you actually reach for? You don't have to build all of this yourself. Here's the latter simplest first. Run one. The minimum. Put idempotency keys on your side affecting calls. Even with no framework at all, That alone stops the double charge on a retry. Rung 2, add a journal and append only log of completed steps so a restart can replay finished work instead of redoing it. Rung 3, and this is where most teams land for serious work, a durable execution engine. Temporal REST8 DbOS. It gives you the journal, the deterministic replay, and the retry machinery out of the box. You write normal looking code and it survives crashes underneath. But the engine is not magic. It still demands the same two things from you. Keep the orchestration deterministic and make the side effects idempotent. The latter changes how much you build. Never the rule. Reach as high as the stakes justify. The whole thing in one breath. The trap, a crash plus a naive Resuméry runs your side effects and double charges. The mechanism, a journal replays finished steps instead of rerunning them, but the engine is only at least once. The test. For every step, can it be safely replayed? Deterministic logic, idempotent side effects. Durability isn't saving your place. It's making every step safe to run again. That's durable execution, agents that survive a crash without redoing their work. If this saved you a double charge, share it with one person building agents. Subscribe to AI Tech Book, one architecture concept, every episode. Thanks for building with us.

Your AI agent runs for an hour — charging cards, sending emails, writing rows — then it crashes. You restart it, and it charges the customer twice. This is durable execution: what it actually takes for an agent to survive a crash WITHOUT redoing its real-world work.

Most takes treat "surviving a crash" as checkpoint + retry — save your place, resume. That misses the hard part: an agent's steps are side effects in the real world, and a naive resume re-runs them. Durable execution isn't save-and-replay — it's making replay SAFE. We build the whole mechanism from the ground up: why replay is dangerous, the exact non-deterministic bug that double-charges a customer, the journal that replays finished steps instead of re-running them, the determinism rule, why every engine is at-least-once (never exactly-once on its own), and how idempotency keys make a retried call charge exactly once at the service. You leave with one test for every step — can it be safely replayed? — and the build-instead ladder: idempotency keys, a journal, or a durable-execution engine like Temporal, Restate, or DBOS.

Grounded in primary sources: Temporal's determinism and idempotency docs, Stripe's idempotency keys, Restate and DBOS on durable execution.

Chapters:
0:00 Intro
0:12 The resume button that lies
1:05 Your steps touch the real world
1:58 The double charge
2:53 Replay, don't re-run (the journal)
3:50 The determinism rule
4:47 The step that crashed mid-action
5:36 The key that charges once
6:33 The one question per step
7:25 What to reach for
8:22 Recap
8:48 Outro

What you'll learn:
- Why "checkpoint + retry" is the wrong mental model for crash-safe agents
- The journal/replay mechanism every durable engine shares
- The determinism rule, and why side effects must move into recorded steps
- At-least-once vs exactly-once, and how idempotency keys close the gap
- The build-instead ladder: idempotency keys, a journal, or a durable-execution engine

If this saved you a double charge, subscribe for one AI architecture concept every episode.

#AIAgents #DurableExecution #Idempotency #AIEngineering #LLM