Discover how to guarantee one-time processing and data consistency in your payment systems using Idempotency and Deduplication.

I have been working in the development industry for quite some time, and most of this experience is in Billing and Payment processors.

Now, as a Developer and Tech Lead, I notice that many developers get stuck or make mistakes because they don’t have experience fixing some very common issues. Often, they have never developed a payment processor and are unaware of a crucial topic: how to prevent duplicate payments or data inconsistency in our databases.

In this article, I will describe and demonstrate two basic techniques used in the industry to avoid these troubles: idempotency and deduplication. I will also explain the difference between these principles, as many developers think they are the same, but I’ll show you why they aren’t.

Let’s imagine we have a simple stateless application with a load balancer in front of it. Normally, the load balancer splits requests across our application instances. In a normal flow, we persist the payment request and publish it to a topic or send an event.

We’ll have a payment request processor that consumes from a topic or queue. It acts as the payment provider (or connects to a 3rd party provider), persists the request type, and publishes the result or returns it to the frontend and hopefully, a successful process.

But imagine a scenario where a user clicks a button twice, and our frontend guardrails fail, sending two payment requests. Let’s look closer at this situation:

User -> Load Balancer -> Payment Request 1, Payment Request 2

When we persist this, we have two payment requests for the same operation. We will publish it twice, process the payment twice, and because of concurrency, we will probably return “success” twice. However, the frontend will likely only show one success message to the User.

Now, you just have to wait for the “war room.” Depending on your context, maybe you shipped 2 products, or maybe you charged the user’s credit card twice. Support will likely alert you, and your product’s KPIs will definitely feel the hit.

So, let’s revisit this scenario to see how to avoid this trouble. First, I won’t dive into transaction locking here; that deserves its own article. If you want to dive into that topic, I suggest researching concurrent transactions, optimistic locking, and pessimistic locking. (I will write an article about this soon).

The Beauty of “One Action, One Return”

When the User clicks the button, we should create an idempotent operation. But what does that mean?

Idempotency

Idempotency means it doesn’t matter how many times I execute a process; the result should be the same. For example, in a REST API, the DELETE endpoint should be idempotent. If we delete a resource and then try to delete the same resource again, the result should return the same HTTP status because the resource was already handled (deleted).

But how can we achieve this in our payment request case? In other words, how can I make many requests, avoid mutating the data incorrectly, and deliver only the one return I expect?

There are many ways, but one of the easiest is for the frontend to generate a unique Idempotency Key when sending the request. This can be a hash or a UUID v4, passed as a header or (my personal preference) directly in the request body.

This way, the Client waits for the response associated with the specific idempotency key created at the moment of the request. This ensures the client side always receives the expected return and allows the system owner to control retries safely. The Client doesn’t need to worry if it is the fifth execution or the first because it will act on exactly the execution it wants.

To guarantee idempotency for the total operation, we should carry the idempotency key through every step, including checking the status of the operation in the database using that key (PENDING or PROCESSED). This guarantees that we will not process a payment request more times than needed.

Crucially, the database column storing this key must have a Unique Constraint. This ensures that even if our application code fails to detect a duplicate due to a race condition, the database engine itself will reject the second insert.

Therefore, we should save the idempotency key in both databases (payment request and payment processor). This protects us even if we publish the message to a topic twice: the payment processor will check the key and process it only once.

Deduplication vs. Idempotency

Once we have this idempotency key and publish the payment request to a topic, the key changes its role slightly and serves another basic technique.

When the payment processor checks the key in the database and realizes we have already processed this payment, we can simply discard the message. We call this technique deduplication.

So, while idempotency means executing multiple times to yield the same result without unintended state mutation, deduplication simply discards subsequent operations. In terms of mutation, neither mutates the state incorrectly, but the difference is that Idempotency returns the same result (often replaying the original success response), whereas deduplication just drops the next operations.

Instead of using only the payment processor’s database, we can use Redis, for example, using a simple SETNX to check if the key already exists. If yes, we can discard the message.

    // A simple conceptual example of the check
    if (redis.setnx(idempotencyKey, "PROCESSING") == 0) {
    // Key already exists, so we define logic to discard or return current status.
    // Of course, avoid literal strings, this is just an example.
    return discardMessage();
    }
    // Don't forget the TTL!
    redis.expire(idempotencyKey, duration);

Just remember to set a TTL (Time To Live) on this Redis key to avoid blocking valid retries indefinitely if your application crashes mid-process.

Conclusion

This article covers the easiest way to avoid duplicate payments in payment systems. There are other techniques I didn’t mention, such as sending an event with the idempotency key directly to the client at the end of the operation, or persisting a hash in Redis to retrieve at the end.

Here you can see a simple example of the final architecture:

I tried to cover the most common scenario. I also didn’t mention how to handle consistency in concurrent database transactions. That can be another article, but if you want to know more now, you can research optimistic locking and pessimistic locking.

I hope you learned something! Feel free to comment with your thoughts or questions. See you in the next article.