Idempotency System Design || Stop Duplicate Processing at Billion Scale || Part 1

Welcome to The Explainer! Today, we're diving into a senior-level, high-level design walkthrough, focusing on one of the absolute most critical foundational architectures in distributed systems, distributed item potency. Look, whether you're a staff engineer trying to harden a massive payment system, or you're just prepping for a really rigorous system design interview, understanding how to safely handle retries and event processing is completely non-negotiable. So we're going to break down exactly how to build a bulletproof correctness layer from the ground up. Think of this agenda as our whiteboard outline for the walkthrough. We'll start with the phantom request problem, build our correctness shield, design a bulletproof dedupe key, compare sync versus async, coordinate our storage, and finally, look at a really common horizontal scaling trap. All right, section one, the phantom request problem. Because to understand the solution, we really need to understand the real-world business damage caused by seemingly harmless retries. Okay, let's dive right into this chaos. You know how it is in a distributed environment. A single, completely innocent user action, like just clicking submit on a checkout page, can literally spawn an absolute storm of phantom requests. Why does that happen? Well, it's usually because of client timeout retries, or maybe impatient double clicks, a worker crashing and triggering a redelivery, or even just a network timeout that happens right after a successful database commit. Without a rock-solid defense mechanism, these phantom requests just repeatedly hammer your domain databases. And the business damage there is devastating. I mean, imagine accidentally charging a customer's credit card twice for rent, duplicating heavy background jobs that completely choke out your compute, or spanning a brand new user with five identical onboarding emails. No good. Because of all that chaos, we absolutely must engineer our systems around this one golden rule. This is our core axiom. One logical operation should produce exactly one intended side effect. This right here is the foundational law for achieving exactly one semantics, and it's going to guide literally every architectural decision we make today. Moving to section two, the cross-cutting correctness shield. So how do we actually enforce that golden rule and solve this chaos? Well, first things first, item potency is not just some random check scattered inside every single endpoint. If you have if duplicate then return logic buried deep inside your core business functions, honestly, your architecture is already bleeding. Instead, item potency needs to be a cross-cutting correctness shield. It's like a dedicated ring that intercepts duplicate requests before they ever even touch your pristine core domain service. This shield owns detecting duplicates, tracking the operation state, managing time to live and lease expiries, and crucially, it seamlessly returns the previous results to the caller. Let's move to the architecture and see how this builds out. The flow here is highly, highly structured. A request comes from the client or producer straight to the API gateway. The gateway forwards it to the domain API service, which then immediately checks in with the independency store. Now, if it's a new request, it gets the green light to mutate the domain DB and write to an outbox table. From there, the outbox relay pushes messages to a durable transport broker like Kafka or SQS, where async workers consume them, check their own Ddupe store, and finally execute the external side effect. It's a beautifully modular, deeply coordinated pipeline. But to make that pipeline work without turning into spaghetti code, we need incredibly strict boundary control. I want to definitively state this. Your API gateway owns authentication, tenant resolution, and forwarding headers. it should never own business item potency decisions. Keep that separation clean. The Domain API service owns executing business operations and calling the item potency module, but it should not scatter raw duplicate logic across its endpoints. And your async workers? They own item potent processing, meaning they absolutely must never blindly execute side effects without first acquiring a duplication lock. Keeping these boundaries rigid is exactly what prevents that architectural bleed we talked about. Okay, Section 3, Anatomy of a D-Duke Key. We've seen the macro level, so let's zoom in on the atomic level to see how we uniquely identify an operation. The reality is you simply can't deduplicate what you can't identify, right? A bulletproof dedupe key precisely defines the boundary of an operation. It's built using a very specific formula. Your tenant ID plus the operation type plus the business resource ID plus a version. So something like tenant-xyz-createorder-order123-v1. The tenant ID prevents any cross-tenant collisions. The operation type separates identical resources across different APIs. The business resource ID is your target entity. And the version? That allows updates to the same entity over time. To really appreciate how good that formula is, let's contrast it with why bad keys fail so spectacularly. If you use only a timestamp or a purely random UUID, you're going to suffer from under-deduplication. Why? Because every single retry looks brand new to the system, so valid retries just drop right through your shield and execute multiple times. On the flip side, if you just use a user ID, You get over deduplication. You literally end up blocking a user from performing two completely legitimate, consecutive, but totally different actions. Your key has to be perfectly balanced. Now, once you have that perfect key, it goes through a very strict lifecycle. It starts as new. When a request actually arrives, it transitions atomically to in progress. If it succeeds, it flips to succeeded. Depending on what happens, it might also hit failed retriable, failed final, or expired. And preventing race conditions here means these must be strict atomic state transitions. You literally cannot have two concurrent requests pushing a new key to in progress at the exact same millisecond. Only one gets to win that lock. To maintain the integrity of that request key lifecycle, your database record needs a few absolutely critical fields. Locked until is necessary so that if a worker crashes while a key is in progress, another worker can eventually jump in and take over when the lease expires. ExpiresAt forces TTL cleanup so your database doesn't just grow infinitely. RequestHash is actually brilliant. It detects if a client uses the exact same idempotency key but maliciously, or even accidentally, changes the JSON payload. And finally, Status obviously tracks your exact position in the state machine. Next up, Section 4, Sync APIs vs. Async Events. Let's shift our focus to the two primary battlefields where this shield actually operates. Now, what's really interesting here is how the fundamental mechanics shift depending on the battlefield. For synchronous APIs, the client provides an item potency key HTTP header. When a duplicate is detected, this system intercepts it and returns the exact same stored payload, like an HTTP 200 or 201. The client just thinks it succeeded normally. But for asynchronous events traversing a queue, the system derives the dedupe key from the event payload itself. And here, on duplicate, the worker just quietly skips execution and acknowledges the message to drain the queue. It doesn't need to return a payload, it just stops the side effect in its tracks. And you know, this distinction also carefully dictates our storage roles. Distributed systems collapse when engineers confuse the fast path cache with a durable system of record. Redis is fantastic for short window, ephemeral fast path lease locks for your async events. But listen to this warning. For synchronous APIs handling critical flows like financial ledgers, payment states, or processed event history, a durable, strongly consistent database like SQL or Spanner absolutely must remain the ultimate source of truth for your item potency state. Which brings us to Section 5, Storage and Outbox Coordination. How do we tie all these states together without causing a catastrophic system split brain? The answer is wrapping everything inside a single atomic database transaction. you execute four strict steps. Step one, insert the item potency key as in progress. Step two, update your core domain entity state, like marking an order as paid. Step three, insert an outbox event like order paid event. And step four, update the item potency key as succeeded along with the final JSON response. And this brilliantly illustrates why this pattern is just non-negotiable. Network failures between database writes and queue publishing are fatal. By coordinating the domain state mutation, the idempotency state, and the outbox event rate inside that one single boundary, we guarantee that all three succeed together or they fail together. An async outbox relay then safely reads the durable outbox table and pushes it to Kafka. There's literally zero risk of your database saying an order is paid, but downstream systems never finding out because the network failed before publishing the event. All right, section six, the horizontal scaling trap. Before we wrap up, we really need to highlight a critical system scale constraint that catches way too many senior engineers off guard. The trap is relying on in-memory deduplication. When you run API servers and workers, they inevitably have to scale horizontally behind a load balancer. If server A processes the first request and stores the key in its local memory, and then a retry comes in, but the load balancer routes it to server B, well, server B's local memory is empty. The phantom request goes right through. This proves that a centralized idempotency or dedupe store, utilizing unique database constraints, is absolutely mandatory to avoid race conditions at scale. To summarize this blueprint, true resilience isn't just one component. It's a coordinated dance. Phase 1, the shield intercepts using a centralized store. Phase 2, the core executes the domain mutation and the outbox write within a single transaction. Phase 3, the relay distributes the event safely to downstream consumers. So the crucial point is this final takeaway equation. You simply cannot guarantee exactly once delivery over a volatile network. But if you combine at least once delivery from your message broker with strict impotent processing at the node level using the blueprints we just discussed, you achieve effectively exactly once business semantics. Your system behaves flawlessly no matter what the network throws at it. But what happens when things go really wrong inside the node itself? What happens when a worker crashes mid-execution and leaves a key stuck in in-progress forever? If your system can't heal itself from that, is it truly resilient? Well, that brings us to the end of part one. Join me for part two, where we're going to dive into the deep end of production-grade recovery, mastering lock leases, automated reconciliation jobs, and dead letter queue handling. You definitely won't want to miss it. Until then, keep building resilient systems.

Design an Idempotency and Deduplication system to stop duplicate processing in distributed systems.
Learn how retries, timeouts, Kafka redelivery, and worker crashes are handled using idempotency keys, dedup stores, Redis, SQL, and the outbox pattern.

In this system design video, we break down Idempotency and Deduplication as a reusable HLD template for building retry-safe distributed systems.

A single user action can create multiple duplicate requests because of client retries, double-clicks, network timeouts, worker crashes, Kafka/SQS redelivery, or webhook duplicates. Without a correctness layer, this can lead to double debit, duplicate job creation, repeated notifications, and inconsistent business state.

In this video, we cover how to design an Idempotency and Deduplication layer that ensures one logical operation produces only one correct side effect.

What you will learn:
- Why distributed systems create duplicate requests
- Idempotency vs deduplication in system design
- Idempotency-Key and dedupKey design
- Request hash validation
- Idempotency store using SQL/Spanner
- Redis fast-path vs durable database source of truth
- Request key lifecycle: NEW, IN_PROGRESS, SUCCEEDED, FAILED_RETRYABLE, FAILED_FINAL
- How to avoid race conditions during retries
- Why in-memory dedup fails with horizontal scaling
- Outbox pattern for reliable event publishing
- Sync API retries vs async event deduplication
- How at-least-once delivery plus idempotent processing gives effectively exactly-once semantics

This video is useful for:
- Senior Software Engineer system design interviews
- Staff Engineer HLD interviews
- Backend distributed systems design
- Payment, wallet, order, coupon, notification, and event-processing systems
- Anyone preparing reusable system design templates

Common interview questions answered:
- How do you design idempotency at scale?
- How do you prevent duplicate payments or duplicate orders?
- How should an idempotency key be stored?
- Should Redis or SQL be used for idempotency?
- How do you handle retry storms and race conditions?
- How do you design deduplication for Kafka/SQS consumers?
- How do you achieve exactly-once behavior in distributed systems?

Watch next on ScalaBrix:
- Multi-Tenant SaaS Architecture HLD Template
- Global Deduplication System Design
- Real-Time Top K at Billion Scale
- Async Processing Architecture Template

#SystemDesign #Idempotency #Deduplication #DistributedSystems #HLD #SystemDesignInterview #BackendEngineering #ScalableArchitecture #scalabrix