The Friday deploy that charged customers twice

It was 4:40 on a Friday. The change was tiny — a one-line tweak to how we called the payment provider, plus a bump to the queue worker's timeout. Small, tested, reviewed. I deployed it and started closing tabs.

At 5:15 the first email came in: "Why was I charged twice?" Then another. Then eleven more. We had double-charged real customers on real cards, and every minute the deploy stayed live, the number grew.

This is the story of that hour, because the bug wasn't really in the one line I changed. The one line just woke up a bug that had been asleep in our payment job since the day it was written.

What actually happened

Our "charge the customer" job called the payment provider, then recorded the charge in our database. Simple. It had run cleanly for months.

The provider's API was occasionally slow — sometimes a call took 30 seconds. Our old worker timeout was 60 seconds, so it never mattered. My deploy lowered the worker timeout to 25 seconds "to fail faster." Reasonable in isolation. Catastrophic here.

Here's the sequence, and it's worth reading slowly:

sec

The job called the provider. At 25 seconds the worker timed out and was killed — before the provider responded. The charge went through anyway, at 29 seconds, but our job was already dead and never recorded it. The queue saw an unfinished job and did exactly what queues are designed to do: it retried. The retry called the provider again. Second charge. Same card.

The job assumed it would run exactly once. Queues never promise that. They promise at least once.

The hour

• 5:15 — First reports. I think it's a support fluke.
• 5:20 — Fourth report. I pull the payment logs and see pairs of charges seconds apart. Stomach drops.
• 5:25 — I roll back the deploy. This stops new double-charges but does nothing for the ones already made.
• 5:30 — We pause the payment queue entirely so no more retries can fire.
• 5:45 — Script written to find every double charge in the window and refund the duplicates.
• 6:30 — Refunds issued, affected customers emailed a real apology before they had to ask.

Rolling back stopped the bleeding. It did not undo the damage. That distinction is the whole reason this was a bad afternoon and not a five-minute fix.

The real fix: make the job idempotent

The timeout change was just the trigger. The actual defect was that our charge job could run twice and produce two charges. The fix is to make the second run a no-op — an idempotency key the provider recognizes:

public function handle(PaymentGateway $gateway): void
{
    // A stable key derived from the thing we're charging FOR,
    // not from the attempt. Same order = same key, every retry.
    $key = "charge:order:{$this->order->id}";

    $gateway->charge(
        amount: $this->order->total,
        source: $this->order->payment_method,
        idempotencyKey: $key, // provider dedupes on this
    );
}

Every serious payment provider supports an idempotency key. You send the same key on a retry, and the provider recognizes it and returns the original charge instead of making a new one. Our first attempt and the retry would have carried the identical key, the provider would have collapsed them into one charge, and no customer would ever have known there was a timeout at all.

We added a second layer too — record "charge started" with the key before calling the provider, so even our own code can check "did I already try this?" and refuse to double-fire.

What else changed

Idempotency was the fix. Two process changes made sure the next one couldn't get this far:

Feature flags on risky changes. The timeout change now would ship behind a flag, off by default, flipped on for 5% of traffic first. A double-charge at 5% is 12 angry emails. At 100% it was hundreds. The flag is the difference between a scare and an incident.

No Friday-afternoon deploys to the payment path. Not superstition — math. The cost of a bug is the size of the blast times how long it stays live. Friday at 4:40 maximizes the second term: everyone's leaving, monitoring attention is at its weekly low, and the person who could fix it fastest is already on the train home. The same deploy Tuesday morning would have been caught in ten minutes by someone actually watching.

The uncomfortable truth is that none of my "small, tested, reviewed" was wrong on its own. The timeout change was sane. The code passed. The review found nothing — because the bug wasn't in the diff, it was in an assumption the diff happened to expose. That's what these incidents almost always are: a latent assumption, sleeping quietly at low volume and short timeouts, waiting for one innocent change to turn the conditions just enough to wake it up.