Exactly Once Delivery Is a Myth Here’s the Truth

Welcome to The Explainer. Today we are going to bust a massive, industry-wide myth about distributed systems. You know, we've all been intuitively led to believe that when we send a message or a request over the internet, it gets delivered exactly once. It makes sense, right? You click send, it arrives. Well, I'm here to tell you, the internet lies to us. Exactly once delivery over a network is a complete myth. And in this explainer, we're going to break down exactly why that is, why exactly once processing is so incredibly hard to achieve, and the brilliant engineering strategies you actually need to build to protect your systems. So let's dive right in. Picture this because I guarantee you've been here. It's a highly relatable, incredibly high stakes scenario. You've just finished a long line with Uber. You open the app, you click pay and the network hiccups. The app just sits there spinning. And eventually, the request times out. So what do you naturally do? You frantically click pay again. Without the right system design running in the background, you've just been charged twice for the exact same ride. And this isn't just theoretical. At scale, these network timeouts happen hundreds of thousands of times a day, creating a massive empathy-inducing risk of double billing for everyday users. The harsh truth of the matter is that exactly one's delivery over a network is just a fantasy. Standard networks generally offer two realities. First, there's at most once delivery. This is essentially a fire and forget method, where a message might just get dropped and never arrive at all. Second, there's at least once delivery. This is where protocols like TCP guarantee delivery through automatic retries, but as a direct result, they might deliver that message twice. They absolutely do not guarantee exactly one semantics. Think about it. A request might actually succeed on the server, but the acknowledgement sent back to the client gets lost in a dropped connection. Because the client never got that confirmation, it automatically retries. And boom, you have a duplicate request. Let's move into section 1, network failure, and look at the underlying physics of the problem. Werner Vogels, the CTO of Amazon, completely hit the nail on the head when he said failure and slowness are normal conditions. This is a completely crucial mindset shift for anyone building systems today. Timeouts, automatic retries, dropped acknowledgments, these aren't rare anomalies that you can just sweep under the rug and ignore. They are architectural primitives. They're literally baked into the very physics of how distributed systems operate. You simply cannot engineer a perfect network. Which brings us to Section 2, the Paradigm Shift towards Smarter Applications. The absolute key to stopping duplicate actions relies on a specific mathematical concept called item potency. Now, the formula f of x equals f of f of x might sound like heavy, complicated math, but the practical definition is wonderfully simple. It basically means an operation produces the exact same result whether you execute it once, twice, or a hundred times. By making our applications idempotent, we completely neutralize the threat of network retries. If a client device or an API gateway automatically retries a failed request, an idempotent server essentially says, hey, I already did this for you. Here is the result, without ever reprocessing the action or charging you twice. This reveals a really strict division of labor between your network and your code. Message queues like Kafka or Amazon SQS default to at least once delivery. They prioritize availability, meaning they will definitely deliver your message, but sometimes they're going to deliver it more than once. Because of that, exactly once processing isn't some magical setting you just toggle on in your network dashboard. It's robust deduplication logic that developers must actively build into their backend to handle those inevitable retries safely. Let's crack open Section 3, the Item Potency Toolkit to see how exactly once processing actually gets built. There are three primary patterns engineers use to construct these safety nets for their data. We have natural idempotency, explicit keys, and conditional updates. Let's isolate each of these for a deeper look into how they function in the wild, starting with the simplest one. First up, natural idempotency. Some operations are inherently safe to repeat without needing any extra databases or infrastructure. Think about standard HTTP verbs like put or delete. If I send a request to set your email address to useradexample.com, I can safely send that request 50 times in a row. The end result is exactly the same. Your email is set to that value. Similarly, deleting a record that has already been deleted just safely succeeds. No complex deduplication tables are required for this stuff. But what about actions that are not naturally safe? Things like charging a credit card or placing a new order. Well, that is where we use explicit keys and duplication tables. In this scenario, the client application generates a unique ID, like a UUID, for the transaction. The server receives this key and stores it in a duplication database. If the network drops and the client retries without exact same key, the server intercepts it. It acts almost like a bouncer at the door. It checks the table, sees the key is already processing or completed, rejects the duplicate request, and simply returns the cached original result. The third path we can take is conditional updates, which is often implemented as optimistic logging. Imagine a collaborative system where multiple devices might try to update an order status at the exact same millisecond. We use version numbers in our database updates to prevent total chaos. The system basically says, update this record to shipped, but only if the version number is currently five. If another request already bumped it up to version 6, the duplicate update safely fails, completely preventing corrupted state machines or lost updates. It's incredibly elegant. All right, let's look at Section 4, Hyperscale Proof, and walk through some massive real-world examples. Take a second to wrap your head around the staggering number, $640 billion. That is Stripe's annual processing volume. When you are moving that much money, the stakes of preventing double charges are... unbelievably high. A single duplicate charge creates customer support nightmares, hefty chargeback fees, and intense regulatory scrutiny. For payment processors like Stripe, IDIpotency keys aren't just a best practice. They are the absolute bedrock of their entire business model. So how does Stripe actually pull it off at that scale? They use a very strict, highly optimized impotent flow. Step one, the client generating the payment must create a unique UUID and send it over in a specific impotency dash key header. Step 2. Stripe's server checks a lightning-fast Redis cache to see if that key already exists. If it does, step 3 kicks in. Stripe instantly returns the cache result of the original transaction without recharging the card. They even go so far as to hash the request parameters to ensure a malicious user isn't trying to reuse the same key for a completely different transaction. But here's the catch, and it's a big one. You can't just store these keys forever. That leads us to a clever optimization hyperscalers use called TTL, or Time to Live. Both Stripe and Uber intentionally expire their stored idempotency keys after exactly 24 hours. Why? Well, because if you process millions of transactions a day, storing every single key indefinitely would lead to terabytes of unbounded database growth. Your systems would eventually grind to a halt. 24 hours is the perfect sweet spot. It's a brilliant hack that's long enough to catch all those immediate network retries, but short enough to keep storage costs and database search latency totally manageable. Uber actually takes this whole concept a step further by using a single item potency key for an entire distributed workflow. When a ride completes, the mobile app generates a single key tied to your specific trip ID. Uber then uses this one key across multiple independent microservices to update the ride status, charge the rider, pay the driver, and send the receipt. If the network fails at literally any point, the app just retries the whole flow with that same key, and the system seamlessly skips the steps it already completed. It guarantees the entire complex workflow is atomic and safe. Let's wrap things up with Section 5, Practical Takeaways, and some architectural lessons you can apply to your own work. Here is the final architectural trade-off you have to consider. Knowing when to use item potency and when to confidently skip it. You should absolutely use explicit item potency keys for high-stakes use cases like financial transactions, payments, and inventory management. But you have to remember that checking a deduplication table adds about 5 to 10 milliseconds of latency to every single request. So for low-stakes scenarios like analytics tracking or application logs where a duplicate entry won't exactly ruin someone's day, it's actually better to just skip it. The extra latency overhead just isn't worth the cost there. Which leaves us with this pretty provocative question for you. Are your system's retries safely designed for the inevitable failure? Remember, exactly once delivery from the network is a complete myth. Failure and slowness are normal conditions. So take a hard look at your own retry logic and your message cues. Are you blindly hoping the network will protect you? Or are you actively building robust, idempotent safety nets to protect your data and your users? Definitely something to think about the next time you click Pay. Thanks so much for joining me on this explainer, and keep building resilient systems.

🚨 Think your backend is optimized? Think again.

In this video, we break down Exactly-Once Delivery Is a Myth (Here’s the Truth) with real production examples, backend performance insights, and distributed systems architecture lessons. 

You’ll learn advanced backend performance strategies, system design trade-offs, scalability patterns, and production debugging techniques used by senior backend engineers.

This video covers:
• Real-world performance bottlenecks
• Distributed systems trade-offs
• Scalability architecture decisions
• Backend optimization strategies
• Production engineering mindset

If you're serious about backend engineering, system design, microservices, Kafka, database optimization, observability, or performance tuning — this series is built for you.

🔥 Subscribe for deep backend performance and system design breakdowns every week.

#BackendEngineering #SystemDesign #DistributedSystems #PerformanceOptimization #Microservices #Kafka #DatabaseOptimization #Scalability #SoftwareArchitecture #TechLeadership