Why Your Rork App's Stripe Webhooks Drop Events Only in Production — Field Notes on Idempotency, Retries, and Out-of-Order Delivery

Layer 2: Make the handler idempotent on event.id

This is the core. Use event.id (evt_...) as the idempotency key and never process the same event twice. Keep a "processed" marker in KV and check for it the moment the event arrives.

async function handleEvent(event: Stripe.Event) {
  const dedupeKey = `webhook:processed:${event.id}`;
 
  // Already handled? Do nothing and return 200 to end Stripe's retries.
  const seen = await KV.get(dedupeKey);
  if (seen) {
    return NextResponse.json({ received: true, duplicate: true });
  }
 
  try {
    await routeEvent(event);
  } catch (err) {
    // On failure, don't write the marker. Return 500 to invite a retry.
    console.error(`processing failed for ${event.id}:`, err);
    return NextResponse.json({ error: 'processing failed' }, { status: 500 });
  }
 
  // Write the marker only after success. TTL longer than Stripe's retry window (up to 3 days).
  await KV.put(dedupeKey, String(Date.now()), { expirationTtl: 60 * 60 * 24 * 7 });
  return NextResponse.json({ received: true });
}

The subtle part is when you write the marker — after the work completes. If you write it first, an event that dies mid-processing gets marked "done," and the retry that follows is rejected and lost forever. The order is: check existence, process, record on success.

Because KV can be eventually consistent across loosely coupled reads and writes, two near-simultaneous deliveries of the same event can still slip through. I make the entitlement grant itself idempotent too — read the current grant state, then write only if it differs — as a second line of defense. Putting idempotency in both the receiver and the business-logic layer, rather than in one place, proved to be the safe choice.

Layer 3: Return 2xx fast, then do the heavy work

Stripe treats a slow response as a delivery failure and retries. If your handler calls several external APIs or waits synchronously to send an email, the response can exceed Stripe's timeout and a retry piles on while you're still working. That's the classic retry storm. Idempotency stops the double grant, but the wasted re-runs burn Workers execution time and pollute your logs.

The fix is to keep only verification and the dedup check synchronous, and push the heavy work past the response with waitUntil.

import { NextResponse } from 'next/server';
 
async function handleEvent(event: Stripe.Event, ctx: ExecutionContext) {
  const dedupeKey = `webhook:processed:${event.id}`;
  if (await KV.get(dedupeKey)) {
    return NextResponse.json({ received: true, duplicate: true });
  }
 
  // Acknowledge receipt, then move the heavy work into the background.
  ctx.waitUntil(
    routeEvent(event)
      .then(() => KV.put(dedupeKey, String(Date.now()), { expirationTtl: 604800 }))
      .catch((err) => {
        // Push failures to a dead-letter queue; the reconciliation job picks them up.
        console.error(`bg processing failed ${event.id}:`, err);
        return KV.put(`webhook:dlq:${event.id}`, JSON.stringify({
          type: event.type, at: Date.now(),
        }), { expirationTtl: 604800 });
      }),
  );
 
  return NextResponse.json({ received: true });
}

This design has a trade-off, though. Once you move work into waitUntil, a failure still returns 200 to Stripe, so you can no longer rely on Stripe's built-in retries. That's why failures go to a dead-letter queue (recorded in KV under a dlq: prefix) for the Layer 4 reconciliation job to recover on its own. Returning 500 synchronously to let Stripe retry, versus acknowledging and retrying yourself, is an either/or. For work like granting access — where a few seconds of delay is fine but a dropped event is not — I chose the latter.

Layer 4: A reconciliation job that fixes reordering and gaps

The final layer is to stop treating the webhook as the single source of truth. Assume reordering and gaps happen, and periodically reconcile against Stripe's current state.

For reordering, the robust move is to not trust the payload when a subscription event arrives, but to retrieve the subscription by id and re-read its current state on the spot.

async function reconcileSubscription(subId: string) {
  // Use the retrieved "current truth," not the status in the payload.
  const sub = await stripe.subscriptions.retrieve(subId);
  const active = sub.status === 'active' || sub.status === 'trialing';
  const customerId = typeof sub.customer === 'string' ? sub.customer : sub.customer.id;
 
  // Read before write here, too (idempotent).
  const current = await KV.get(`entitlement:${customerId}`);
  const next = active ? 'pro' : 'free';
  if (current !== next) {
    await KV.put(`entitlement:${customerId}`, next);
    console.log(`entitlement ${customerId}: ${current} -> ${next}`);
  }
}

Now, even if updated arrives before created, you always converge to "Stripe's current state," and the stale-overwrite accident disappears.

To catch gaps themselves, I run a once-a-day reconciliation job via Cron Triggers. It diffs recently updated subscriptions and the events sitting in the dead-letter queue against Stripe, and fixes anything that drifted. After moving to "webhook as primary, reconciliation as insurance," billing-related "it just doesn't match" reports dropped to roughly zero.

# wrangler.toml
[triggers]
crons = ["0 18 * * *"]  # once a day, sweep up anything that was dropped

Where to start

You don't need all of this at once. If you want an order, start with Layer 2 — event-ID idempotency. Double-granting does the most damage and is the hardest to notice. After that, add Layer 3 if retries are eating execution time, and reach for Layer 4's retrieve-based reconciliation early if you handle subscription lifecycles. That sequence covers most of the pain.

Stabilizing webhooks comes down to one idea: don't trust what arrives too much. A dropped delivery costs you less, in practice, than a delivery that arrives duplicated, reversed, or delayed and quietly drifts your state. If you're stuck at the same spot, I hope this gives your investigation a place to begin.