Users Are Getting Duplicate Payments. Find the Bug.

Support forwards three tickets: customers charged twice for the same order, minutes apart, same amount. Someone suggests disabling the pay button after the first click. That will not fix it, because the duplicate almost certainly did not come from a second click.

The correct framing

A duplicate payment is what happens when a retry meets a non-idempotent write. Somewhere in the chain, something timed out or failed ambiguously, something else retried, and the second attempt had no way to recognise the first.

The root cause of the ambiguity is unavoidable: when a request times out, the client cannot distinguish "the charge never happened" from "the charge happened and the response was lost." Both look identical. Retrying is correct behaviour — it is the only way to recover from the first case. So the server must be able to recognise the retry. Preventing retries is not an option; making them safe is.

Client                    Your API                 Payment provider
  │  POST /pay                │                            │
  ├──────────────────────────▶│  charge()                  │
  │                           ├───────────────────────────▶│
  │                           │                       [CHARGED ✓]
  │                           │◀───── 200 ───────────────  │
  │   ✗ timeout at 10s        │                            │
  │   (response never arrives) │                            │
  │                           │                            │
  │  POST /pay  (retry)       │                            │
  ├──────────────────────────▶│  charge()  again           │
  │                           ├───────────────────────────▶│
  │                           │                       [CHARGED ✗✗]

The client did nothing wrong. The endpoint was not idempotent.

Where the second charge comes from

Notice how many of these are outside your application code, and that webhook redelivery is not an edge case — it is the documented, intended behaviour of every payment provider. If your webhook handler is not idempotent, duplicates are certain, not possible.

The read-then-write race

This is the bug hiding in most "we already handle duplicates" codebases. The check and the write are two separate operations, so two concurrent requests both pass the check.

// A check like this provides no protection under concurrency
const existing = await db.payments.findOne({ orderId });
if (existing) return existing;                 // ← both requests see null

const charge = await gateway.charge({ amount, source });   // ← both charge
await db.payments.insert({ orderId, chargeId: charge.id });

// Timeline:
//  T1 reads: none        T2 reads: none
//  T1 charges ✓          T2 charges ✓        two charges, two rows

Making the window smaller does not fix it; only the database can arbitrate. You need a unique constraint, and you need to claim it before talking to the payment provider — so that the loser of the race discovers it lost while no money has moved yet.

How to actually find it

1. Get the two charge records from the provider for one affected customer. Compare timestamps, the idempotency key each carried (if any), and the request ID or user agent. The gap tells you a lot: 200ms apart is concurrency; 10 seconds apart is a client timeout retry; hours apart is a cron job or a webhook.
2. Trace both requests in your logs by order ID. Did your API receive two requests, or one request that called the gateway twice (a retry inside your own HTTP client)? This single question splits the investigation.
3. Check your HTTP client configuration. Many libraries and service meshes retry idempotent-looking failures, including POSTs on connection errors, by default. This is the sneakiest source because nothing in your code looks like a retry.
4. Check the queue. If charging happens in a consumer, look for redeliveries: did the consumer crash or exceed its visibility timeout after charging but before acknowledging?
5. Check the webhook handler. Look for the same provider event ID processed twice. If you have a "create payment record on webhook" path and one on the API path, they may also be duplicating each other.
6. Look for the missing constraint. Inspect the schema. If there is no unique index on an idempotency key or on (order_id, attempt), you have found the bug regardless of what else is true.

The fix: idempotency keys done right

Four rules, in order of importance.

1. The client generates the key, and reuses it across retries

The key must be created once for the user's intent and sent unchanged on every retry. A key generated per HTTP request is worthless — each retry gets a new one and every attempt looks new. This is the mistake that quietly defeats otherwise-correct implementations.

2. Claim the key in your database before charging

-- Schema
CREATE TABLE payment_attempts (
  idempotency_key text PRIMARY KEY,
  order_id        uuid NOT NULL,
  user_id         uuid NOT NULL,
  amount_minor    bigint NOT NULL,
  currency        text NOT NULL,
  status          text NOT NULL,          -- initiated|succeeded|failed
  provider_charge_id text,
  response_body   jsonb,
  created_at      timestamptz NOT NULL DEFAULT now()
);
CREATE UNIQUE INDEX ON payment_attempts (order_id) WHERE status <> 'failed';

async function pay({ idempotencyKey, orderId, amountMinor, currency, source }) {
  // 1. Claim the key. The database decides who wins.
  const claim = await db.query(
    `INSERT INTO payment_attempts (idempotency_key, order_id, user_id,
                                   amount_minor, currency, status)
     VALUES ($1,$2,$3,$4,$5,'initiated')
     ON CONFLICT (idempotency_key) DO NOTHING
     RETURNING id`,
    [idempotencyKey, orderId, userId, amountMinor, currency],
  );

  if (claim.rowCount === 0) {
    // We lost the race, or this is a retry. Return the first result.
    const prior = await db.oneOrNone(
      "SELECT * FROM payment_attempts WHERE idempotency_key = $1", [idempotencyKey]);

    if (prior.status === "initiated") {
      // In flight elsewhere. Do NOT charge. Tell the client to poll.
      return { status: 409, body: { state: "processing", retryAfter: 2 } };
    }
    // Verify the request matches — same key, different amount is an error
    if (prior.amount_minor !== amountMinor) return { status: 422, body: { error: "key_reuse_mismatch" } };
    return { status: 200, body: prior.response_body };
  }

  // 2. We own the attempt. Pass the SAME key to the provider.
  const charge = await gateway.charge({
    amount: amountMinor, currency, source,
    idempotencyKey,                        // provider dedupes too — belt and braces
  });

  // 3. Record the outcome.
  await db.query(
    `UPDATE payment_attempts
        SET status = $2, provider_charge_id = $3, response_body = $4
      WHERE idempotency_key = $1`,
    [idempotencyKey, charge.ok ? "succeeded" : "failed", charge.id, charge],
  );

  return { status: 200, body: charge };
}

3. Pass the key to the provider too

Stripe, Razorpay and the rest all support idempotency keys. This protects the case where your process dies after calling the provider but before recording the outcome — the retry reaches the provider, which returns the original charge instead of creating a new one. Two independent layers of protection, and you need both, because the gap between "charged" and "recorded" is exactly where crashes hurt.

4. Handle the in-flight case explicitly

The nastiest state is a retry arriving while the first attempt is still talking to the provider. You must not charge, and you cannot return a result you do not have. Return a "processing" response and let the client poll, and have a sweeper that resolves attempts stuck in initiated by querying the provider for that key. Treating this state as "probably fine, just charge" is how the bug reappears months later.

Webhooks and redelivery

Providers deliver at least once and will resend until you return 2xx. Every handler needs the same treatment:

POST /api/public/payments/webhook

1. Verify the signature FIRST, before parsing or trusting anything.
2. Insert the provider's event id into processed_events with a unique
   constraint. On conflict: return 200 immediately, do nothing else.
3. Process the event inside a transaction, keyed by the payment attempt.
4. Return 200. Any non-2xx guarantees redelivery.

Also: events can arrive OUT OF ORDER. A 'payment.failed' for an old
attempt must not overwrite a later 'payment.captured'. Guard state
transitions with the state machine, not with last-write-wins.

And do not do slow work inside the handler — verify, record, enqueue, return 200. A handler that times out gets redelivered, which is how one event becomes five.

A payment state machine

Duplicates thrive where state is a boolean. Make the legal transitions explicit and reject everything else; illegal transitions become alerts instead of silent double charges.

initiated ──▶ authorized ──▶ captured ──▶ refunded
    │              │              │
    │              └──▶ voided    └──▶ partially_refunded
    └──▶ failed

Rules:
  - Only 'initiated' may call the gateway.
  - 'captured' is terminal for charging: no path returns to charging.
  - Every transition is a conditional UPDATE:
        UPDATE ... SET status='captured' WHERE id=$1 AND status='authorized'
    Zero rows affected means someone else already did it — that is a
    signal to investigate, not to retry.
  - Store amounts as integer minor units. Never floats.

Reconciliation and refunds

Prevention is not enough for money; you also need detection. Two jobs, both non-negotiable:

• Daily reconciliation. Pull the provider's settlement report and compare against your ledger. Flag charges the provider has that you do not (you charged and lost the record), charges you have that the provider does not (phantom records), amount mismatches, and — the case here — multiple successful charges for one order.
• Stuck-attempt sweeper. Anything in initiated for more than a few minutes gets resolved by querying the provider with its idempotency key, then moved to a terminal state.

And an alert on the specific invariant: more than one successful charge per order. That query is trivial and should page someone, because customers noticing before your monitoring does is the real failure. Refund automatically where policy allows — a fast, proactive refund with a message costs far less trust than a support queue.

If the payment spans multiple services (reserve inventory, charge, create shipment), the coordination problem is a distributed transaction; see the saga pattern for compensating actions, and the outbox pattern for publishing payment events without losing them.

Checklist

• Client generates one idempotency key per intent and reuses it across all retries.
• Unique constraint on that key, claimed before calling the provider.
• Same key passed to the payment provider.
• Stored response replayed for duplicate requests; explicit "processing" reply for in-flight.
• Webhooks: verify signature, dedupe on the provider event ID, return 2xx fast.
• State machine with conditional updates; no transition from a terminal state back to charging.
• Integer minor units for money; never floats.
• Disable automatic retries in HTTP clients for payment calls, or ensure they carry the key.
• Daily reconciliation against the provider, plus a stuck-attempt sweeper.
• Alert on more than one successful charge per order.
• Button disabling in the UI — last, and only as polish.

How to answer this in an interview

"My first assumption is that it's a retry, not a double click. When a request
 times out the client can't tell whether the charge happened, so retrying is
 correct — the endpoint just wasn't idempotent.

 To find it I'd pull both charges from the provider and check the time gap:
 200 milliseconds means concurrency, ten seconds means a client timeout retry,
 hours means a cron job or webhook redelivery. Then I'd check whether my API got
 two requests or one request that called the gateway twice, because HTTP clients
 and meshes retry POSTs on connection errors by default.

 The likely bug in the code is a read-then-write check: find by order id, if
 none then charge. Two concurrent requests both read none and both charge. Only
 a unique constraint fixes that, and it has to be claimed before the gateway
 call so the loser finds out while no money has moved.

 So: client-generated idempotency key reused across retries, INSERT ... ON
 CONFLICT DO NOTHING to claim it, pass the same key to the provider so it
 dedupes too, store the response and replay it for duplicates, and return a
 'processing' 409 if the first attempt is still in flight rather than charging.

 Then detection, because this is money: a state machine with conditional
 updates, daily reconciliation against the provider's settlement report, and an
 alert on more than one successful charge per order."

Summary

• Duplicate charges come from retries — client, HTTP library, queue, or webhook — not double clicks.
• A timeout is ambiguous by nature, so retries are correct; the server must recognise them.
• Read-then-write checks fail under concurrency; only a unique constraint arbitrates.
• Claim the idempotency key before charging, and pass it to the provider as well.
• Handle the in-flight retry explicitly instead of charging again.
• Webhooks are at-least-once and out-of-order: dedupe on event ID, guard with a state machine.
• Reconcile daily and alert on more than one successful charge per order.

Part of Top 10 System Design Interview Questions.