Idempotency System Design || Stop Duplicate Processing at Billion Scale || Part 2

Welcome back to The Explainer. Today we're diving into part two of our series on idempotency and deduplication. If you joined us for part one, you know we covered the theory. Today, we transition from that theory of at least once delivery into a concrete production-grade playbook for achieving exactly once business behavior. Think of this as your high-level system design masterclass. Grab your notepads because we're getting right into the architecture. But first, let's do a really quick 20-second recap of our fundamental takeaway from part one. We already learned that distributed systems usually provide at least once delivery. So duplicate requests and duplicate events, they're totally normal. Magical exactly once delivery across the network simply doesn't exist. So the goal is not magical exactly once delivery. The goal is effectively exactly once business correctness. And we achieve that using robust defense mechanisms like item potency keys, dedupe keys, durable stores, outbox patterns, and background reconciliation. Alright, now let's make this production great. Here are the six stages of our playbook. We'll start with system foundations, move to the request hash defense, look at partitioning strategies, tackle race conditions, walk through the master blueprint, and finally, handle downstream chaos and observability. Okay, let's jump straight into section one, system foundations. The very first step for any architect is isolating the crucial operational split between synchronous APIs and asynchronous events. It impotency is specifically for synchronous API retry safety. This is where we must return the exact same response back to the client. Deduplication, on the other hand, focuses on asynchronous event suppression, where we just want to safely drop duplicate work. Understanding this split dictates exactly how you're going to build out your system. And we have to apply those concepts based on the criticality of the flow itself. Keep in mind that critical operations like payment debits, wallet transfers, coupon consumption, or inventory reservations strictly require strong consistency. Duplicate side effects there are completely unacceptable. But for non-critical operations, things like generating a thumbnail, ingesting analytics, or refreshing a cache, for those, your system can safely tolerate eventual consistency. Moving right along to Section 2. The Request Hash Defense When we look at our perimeter defenses, we have to talk about why an idipotency key submitted by a client is fundamentally insufficient all on its own. The golden rule here is that your idipotency key must be scoped securely, but crucially, it must be paired with a hashed value of the entire request payload. Why? Well, to ensure bad actors or buggy front-end clients aren't just recycling keys for entirely different operations. So let's look at a payload mismatch scenario. Imagine request A arrives to charge 100 US dollars. We hash that payload, let's call it hash X, and we tie it to the item potency key. A minute later, a buggy client sends request B, trying to reuse that exact same key, but this time the payload says 500 US dollars. That generates hash Y. Our system detects that hash mismatch and immediately rejects request B with an HTTP 409 conflict. Boom! We just prevented massive financial damage. Behind the scenes, we manage these requests using five explicitly defined item potency states. In progress means the operation is currently being processed and we're locking concurrent attempts. Succeeded means we just replay the stored response. Failed retriable means it's a temporary failure, so we allow a retry. Failed final is a permanent failure. We return the exact same error and block future retries. And expired means the key is outside our time to live window. Oh, and any unknown state, that should always be handled super conservatively. All right, section three, partitioning and strategy trade-offs. With the keys secured and the states managed, a senior engineer's true test is evaluating how to partition them across the infrastructure stack. Not all data stores are created equal. We might use Redis partitioning for fast, short window deduplication. We use Kafka partitioning to preserve strict processing order for async events. But OpenSearch, OpenSearch carries a very specific absolute restriction. I'm going to say this loud and clear. Warning. Never use a search index for correctness checks. Because search clusters are inherently eventually consistent, they are strictly for debugging. If you try to dedupe a payment based on an elastic search query, I guarantee you, you are going to double charge a customer the second replication lag happens. Just don't do it. So because of those restrictions, we have to strategically select our deduplication method based on the exact use case. Let's look at the four main strategies. First, client provided keys. These are best for synchronous APIs, payments, and order creation. but the server has to validate the request hash. Second, server-generated keys. Ideal for queue consumers, webhooks, and search indexing, but it's tricky to assign the correct business identity. Third, domain-level unique constraints. This is best for ledgers and wallet records. It's your strongest fallback because it prevents duplicates even if the item potency layer completely fails. And finally, Redis-only dedupe. Fast and cheap, but highly risky for critical flows because it's not transactionally tied to your database and can fail during a TTL expiry or a node failover. This brings us to Section 4, Mitigating Distributed Race Conditions. Now we enter the crucible. This is where you earn your architecture stripes. We need to stress test our logic against four distinct failure modes that distributed network execution practically guarantees will happen in fractions of a millisecond. If you're prepping for a system design interview, this is the exact workflow you want to memorize. Let's break them down. First up is simultaneous arrival. Imagine a user aggressively double clicks the submit order button. Two identical network requests hit your API at the exact same millisecond. They both check the database, see no key, and try to write. Our mitigation here? The database unique constraint. The database intercepts the concurrent insert, allows the first one through, and makes the second insert safely air out. Second is the client timeout. SER execution succeeds perfectly. The payment goes through, but then the mobile connection drops. right before our HTTP 200 OK reaches the user. Naturally, the user's app automatically retries. Our system mitigation is to intercept that automated retry using the idiopuntency key, look up the succeeded state, and return the cached success payload without hitting the backend a second time. Third is the pre-side effect crash. A worker successfully claims an idempotency key, marking it in progress, but then the server literally catches on fire and dies before executing any of the business logic. That key is now totally stuck. Our mitigation is setting a locked until lease on the database row. A background reconciliation cron job regularly sweeps up those stale leases, resets the state, and unblocks a safe retry. And finally, the absolute trickiest one. the post-side effect crash. The worker executes the task, say sending an email or charging a credit card, but it crashes the microsecond before marking our database key as succeeded. When the retry comes in, we don't actually know if the side effect completed. Here we rely entirely on the external provider's own message ID at impotency. We pass our unique key over to Stripe or SendGrid so their system blocks the inevitable redelivery. All right, section five, the master architectural blueprint. Having successfully navigated all those brutal race conditions, we can confidently piece together the overarching high-level design. This cohesive blueprint handles both synchronous API retries and asynchronous event duplications simultaneously. Step 1 handles the hash validation right at the API gateway. Step 2 uses Redis for a fast, optional short window check. Step 3 hits the durable, ACID-compliant database using unique constraints. Step 4 utilizes the outbox pattern to guarantee eventual async publishing to our workers. Step 5 is the worker-side dedupe tables. And Step 6 is our background cron reconciler sweeping through and cleaning up any messes. It's a thing of beauty. Finally, Section 6, Downstream Chaos and Observability. Listen. Our blueprint isn't completely production-grade until we properly account for what happens when our downstream providers inevitably collapse. We must build in severe resilience. We use automated circuit breakers and dead-letter cues to route failures gracefully, rather than hammering a down system and creating partial, broken side effects. Simultaneously, we have to continuously monitor our observability signals. Because your item potency check is in the critical path of literally every request, you need a dashboard ensuring your check latency stays under a strict P99 of 20 milliseconds. Otherwise, your DDo player just became your system's biggest bottleneck. We've covered the foundational split, the hash defenses, partitioning trade-offs, and race conditions, giving you a complete production-ready playbook. As you take this explainer back to your own teams to review your current architecture, I want to leave you with one critical question to ask your engineers. Does your system actually fail closed when the durable item potency store goes down? Because if it fails open, you aren't protecting anything at all. Thanks for joining and keep building resilient systems.

In this system design video, we break down Idempotency and Deduplication as a reusable HLD template for building retry-safe distributed systems.

A single user action can create multiple duplicate requests because of client retries, double-clicks, network timeouts, worker crashes, Kafka/SQS redelivery, or webhook duplicates. Without a correctness layer, this can lead to double debit, duplicate job creation, repeated notifications, and inconsistent business state.

In this video, we cover how to design an Idempotency and Deduplication layer that ensures one logical operation produces only one correct side effect.

What you will learn:
- Why distributed systems create duplicate requests
- Idempotency vs deduplication in system design
- Idempotency-Key and dedupKey design
- Request hash validation
- Idempotency store using SQL/Spanner
- Redis fast-path vs durable database source of truth
- Request key lifecycle: NEW, IN_PROGRESS, SUCCEEDED, FAILED_RETRYABLE, FAILED_FINAL
- How to avoid race conditions during retries
- Why in-memory dedup fails with horizontal scaling
- Outbox pattern for reliable event publishing
- Sync API retries vs async event deduplication
- How at-least-once delivery plus idempotent processing gives effectively exactly-once semantics

This video is useful for:
- Senior Software Engineer system design interviews
- Staff Engineer HLD interviews
- Backend distributed systems design
- Payment, wallet, order, coupon, notification, and event-processing systems
- Anyone preparing reusable system design templates

Common interview questions answered:
- How do you design idempotency at scale?
- How do you prevent duplicate payments or duplicate orders?
- How should an idempotency key be stored?
- Should Redis or SQL be used for idempotency?
- How do you handle retry storms and race conditions?
- How do you design deduplication for Kafka/SQS consumers?
- How do you achieve exactly-once behavior in distributed systems?

#SystemDesign #Idempotency #Deduplication #DistributedSystems #HLD #SystemDesignInterview #BackendEngineering #ScalableArchitecture #scalabrix