Your Payment Can't Be Rolled Back | Saga Pattern Explained

Every engineer's first love is the database transaction, do 5 things and if anything breaks, one command, roll back and the universe is restored. Here is the uncomfortable truth. That guarantee dies the moment step 2 lies in somebody else's computer. Look at the screen. Transfer money between 2 banks. The debited bank A succeeds. The credited bank B fails. You cannot roll back the debit, it is not your database and the money really left. You cannot roll back a sent email or a charged card either. The outside world has no undo button. It only offers new operations. The pattern that fixes this is from 1987. It is called the saga. And two weeks ago Cloudflare shipped it as a first class primitive. First question. Why not just stretch the transaction across the services? There is a classical answer and its ghost haunts every distributed systems interview. Two phase commit. A coordinator asks every participant to prepare, do the work, hold the locks, promise you can commit and once everyone votes yes, phase 2 says commit everywhere, atomicity, across machines, on paper. In practice, it stops at your organization's boundary. For three compounding reasons. First, everyone waits for the slowest participant while holding locks. You have coupled the availability of all systems into one chain. Second, the coordinator is a single point of failure of the worst kind. If it dies between phases, participants are stuck in limbo holding locks they cannot release, blocked, sometimes for as long as the coordinator stays down. Third, and this is the killer, two-phase commit requires every participant to expose, prepare and to hold locks on demand. The payment processor does not. The airline's booking API does not. The other bank certainly does not. The interfaces of the real world are commit only, you call them, the thing happens and the response is your receipt. No amount of protocol cleverness upstream can conjure a prepare phase out of an API that does not offer one. So atomicity in its strict form is simply off the menu. What is the correctness model that works with commit only steps? The saga pattern was named in a 1987 paper by Garcia, Moulina and Salem, originally about long-lived transactions inside a single database and it aged into the standard answer for distributed workflows. The move is to stop pretending you have one atomic transaction and admit you have a sequence of local ones. Debit bank A, a real transaction, committed immediately. Credit bank B, another, send the confirmation, a third. No global locks. No coordinator holding anyone hostage. Each step is fast, local and final. Correctness is recovered through a contract. Every step in the saga is paired. At design time, with a compensation, a new forward operation that semantically reverses it. The compensation for a debit is a credit back to the same account. The compensation for a booked seat is a cancellation. Under this contract, a saga has exactly two legal endings. The happy path, all steps complete. The unwind, some step fails and the saga runs the compensations for everything that already completed. In reverse, returning the word to a state that is morally not bitwise, equivalent to never having started. Cloudflare's post compresses it into one line. A saga is the pairing of an operation and its compensation logic. But that word semantically is carrying a lot of weight. What exactly makes a good compensation? First, a compensation is not a restore. The debit happened. It is in bank A's ledger forever. The compensation is a brand new operation, a credit for the same amount, whose business meaning cancels the first. Ledger shows both entries, the book's balance, history is not rewritten, it is answered. That is why the patent survives contact with commit-only APIs. Compensations only need the same interface. The forward step used. Second, order matters. The unwind runs in reverse. Because later steps routinely depend on earlier ones. You must cancel the shipment before you refund the order it belongs to. Undo the newest thing first, walk backward to the oldest. Third, and this is where hand-rolled sagas quietly rot. Compensations run at the worst possible moment. Something already failed. So they themselves will be retried, across timeouts and clashes. Retried operations must be idempotent. Running place must equal running once. In practice that means every compensation carries an idempotency key. Cloudflare's example is literally the transfer id suffixed with rollback debit account. A. So the bank can recognize a duplicate refund attempt and swallow it. Miss that detail and your failure handler becomes the incident, the double refund. So we have steps, compensations, ordering, idempotency. Now the question the hold pattern secretly hinges on, when things explode, who remembers what already happened? Here is the sentence from Cloudflare's post that deserves a frame on the wall. When a multi-step application fails halfway through, the hardest part is often not knowing that it failed. It is knowing what already happened and what needs to happen next. Sit with that. To unwind correctly, you need an exact trustworthy answer to which step started, which finished, with what results, which compensations are registered and which have already run. And you need that answer after the process. Orchestrating the saga has itself died, because the crash that broke your transfer is not polite enough to spare the coordinator. A saga without durable memory is a promise written in RAM. This is why hand-rolled sagas decay into status columns, sweeper crons, and tribal runbooks. Teams keep reinventing a crash-proof memory of progress, one incident at a time. Regular viewers should feel the click. A durable, ordered, crash surviving record of every step start and completion is precisely the event history from our Durable Execution lecture, the journal that outlives the process. Sagas are not a competitor to Durable Execution, they are its most natural feature. Journal the forward steps and the Unwinder can always reconstruct what happened and what it owes the world, which is exactly the foundation Cloudflare built on. What does that look like as an actual API? The evidence slide Cloudflare verbflows is a durable execution engine, code as steps, each step journaled and in late June it gains saga rollbacks as a primitive. The API shape is the pattern from this lecture. Verbatim When you declare a step, you attach its compensation as metadata. Right there. Step.do Debit bank A. The function that debits and an options object carrying rollback. An async function that credits the money back. Two details show the design was earned. First The rollback handler receives the recorded output of its forward step, the compensation credits using the transaction ID, the debit returned, pulled from the journal, not from fragile closure state. Second, they thought hard about ordering under parallelism. Steps can run concurrently and completion order can differ from start order. So unwinding by reverse completion would make the rollback sequence non-deterministic, different on every failure. Workflows unwinds in reverse step start order. Deterministic, reproducible. debuggable no matter how the async thus settled. They also rejected the pretty fluent version. Dot rollback chained after the step because it broke promise, pipelining and step timing semantics, an API where failure handling is declared beside the operation, not scattered in cache blocks three files away. But declaring rollbacks is the easy half. What makes them actually run after a real crash? Under the hood, rollback rides the machinery the engine already trusted. The persisted step history answers authoritatively the three questions from slide 5, which step started, which finished and with what output which rollback handlers were registered. In the common case, the engine simply invokes the handler stubs it already holds. In the recovery case, The orchestrating machine died mid saga. A fresh worker replays the journal and replay rebills the missing rollback handlers without re-executing the original side effects because every forward result is injected from the history rather than re-earned. The debit is not refired to rediscover its transaction ID. The ID is read from the log and the compensation gets exactly what it needs. Then the detail that separates a primitive from a code snippet, compensations run with the same operational guarantees as forward steps, retries with policy, timeouts, lifecycle events, logs and a final recorded outcome. Your refund is not a fire and forget apology, it is a durable step that will be retried through crashes until it succeeds and whose success is itself journaled. Failure handling with the same rigour as the happy path that is the standard hand rolled sagas almost never reach. So should every workflow you own become a saga tomorrow? No and the caveats are load-bearing. The classical objection to sagas is real, no isolation. A database transaction hides your intermediate states. A saga performs them in public. Between the debit and the unwind, bank ace customer sees money gone that will later reappear. The mature response is to design states that are safe to be seen, the order is pending, the funds are held, not taken, the seat is reserved, with expiry. Intermediate states become part of your product's vocabulary instead of a glitch user's screenshot. Second caveat, compensation is not always the right failure strategy. Often the better saga is the one that refuses to go backward. If the failing step can be retried into success, and on a durable execution engine, retries are cheap and automatic. Driving the workflow forward to completion beats unwinding it. Reserve compensations for steps where waiting is not an option and forward repair is impossible. Third, some actions have no compensation. The email is read, the webhook fired, the SMS arrived. You handle those with ordering, put the irrevocable step last after everything failure prone has succeeded or you accept the leak explicitly and log it, rather than pretending a compensating apology email is an undo. Pattern, machinery, price. What is the checklist you carry back to your own systems? The Portable Checklist 5 rules 1. Pair every step with its undo at design time. The moment you write a forward operation, write down what would cancel it. If you cannot say what the compensation is, you have discovered an irreversible step and that discovery belongs in design review, not in an incident. 2. Compensate semantically, never bitwise. You are not restoring a snapshot. You are issuing a new operation whose business meaning cancels the old one. Both entries stay in the ledger and the books balance honestly. 3. Unwind in reverse start order. Later steps lean on earlier ones and start order is the only order that stays deterministic when parallel steps finish shuffled. 4. Idem potency keys on both directions. Forward steps retry, compensations retry harder, every operation must survive running twice or your failure handler will one day be the incident. 5. General progress before trusting it. The unwind is only as good as its memory of what happened. Persist every step start and finish somewhere no crash can reach or run on an engine that does it for you. 5. Rules 1. Theme Failure paths deserve the same engineering as success paths. Let's land a whole story. Landing it When a workflow spans services, the atomic transaction you were promised is gone. The outside world is commit only, holds no locks, and offers no prepare phase. So two-phase, commit stops at the boundary. The saga is the honest replacement, a sequence of local, immediately committed steps, each paired at design time with a compensation, a new forward operation that semantically cancels it, and a contract of two legal endings. Everything completes. Your everything completed gets unwound, newest first, with idempotency keys making every retry safe. The genuinely hard part is not the compensations, it is remembering, across crashes, what already happened and what is owed, which is why sagas are not a rival to durable execution but its natural feature. And why Cloudflare bridge rollbacks directly into workflows, compensation declared beside the step, Unwind in deterministic reverse start order and every rollback running with retries, timeouts and a journaled outcome of its own. If you missed our Durable Execution lecture, it is the prequel to this one, the journal that makes all of this memory possible. The discipline to keep, design the undo when you design the do. Next week we open a new set of systems. Same rule as always, concepts first, receipts from the source.

The database transaction is one of the greatest abstractions in software engineering.

Run five operations. If one fails, everything rolls back as if nothing happened.

But the moment those five operations span multiple services, databases, or APIs, that guarantee disappears.

You can't roll back an email.
You can't un-send a payment.
You can't ask another team's service to pretend nothing happened.

That's where the Saga Pattern comes in.

Instead of one giant transaction, a Saga is a sequence of local transactions connected by compensating actions. In this lecture, we build the intuition behind Sagas from first principles, explain why Two-Phase Commit (2PC) breaks down in modern distributed systems, and look at how production workflow engines embrace compensation instead of pretending distributed rollback exists.

Chapters:
0:00 You Can't Roll Back an Email
0:47 Why Two-Phase Commit Stops at the Service Boundary
2:00 The Saga: A Transaction Made of Transactions
3:18 Compensation Semantics: Undo Is a New Forward Operation
4:40 The Actually Hard Part: Knowing What Already Happened
5:56 Case Study: Rollbacks as a Primitive in Cloudflare Workflows
7:16 Case Study: Compensation as a First-Class Step
8:34 Where Sagas Fit: and the Price of Admission
9:50 The Design Rules: Stealing the Pattern Honestly
11:02 Key Takeaways: Design the Undo

Code That Can't Die | Durable Execution & Temporal Explained: https://youtu.be/8wsoo6gp6eM?si=j9K7TRbNY3_ig0io


Sources & further reading:
• Cloudflare Blog — "How we built saga rollbacks for Cloudflare Workflows":  https://blog.cloudflare.com/rollbacks-for-workflows/
• Cloudflare changelog — "Rollback support now available in Workflows" : https://developers.cloudflare.com/changelog/post/2026-06-05-saga-rollbacks/
• Garcia-Molina & Salem, "Sagas", SIGMOD 1987: https://dl.acm.org/doi/10.1145/38713.38742