Making Silent Failures Visible Across 6 Rork Apps: An Early-Warning Design Note for Non-Crash Degradation

Three design principles — Beacons, timeouts, reverse-aggregation

The implementation collapsed into three principles.

• Beacons (lightweight events): Always emit a single event for "user tapped the ad button" or "user entered checkout" — an intent marker that exists before any failure could occur
• Timeout judgement: For every Beacon that has a successful counterpart, flag the ones where the counterpart never arrives within N seconds, server-side
• Reverse aggregation: Pull AdMob / RevenueCat reports via Cloudflare Workers Cron and reconcile against Beacon counts to derive the missing rate

Doing the timeout judgement client-side alone is a trap. Network-flaky users will trigger false positives, and you lose the cross-app view that matters when you run 6 apps. I keep the aggregation layer deliberately separate from Firebase to avoid mixing billing-affecting events into Crashlytics' pricing tier.

The Cloudflare Worker that receives the Beacons

The Worker is tiny — one POST endpoint, plus an hourly Cron that scans KV and emits "missing counterpart" lists. The production skeleton is below.

// worker/beacon.ts
import type { ExecutionContext, KVNamespace } from "@cloudflare/workers-types";
 
type BeaconEvent = {
  app: string;            // "wallpaper-a" etc — 6 app identifiers
  kind: string;           // "ad-tap" / "rwd-watched" / "purchase-tap" etc
  corrId: string;         // ID that ties an expected pair together
  ts: number;             // ms epoch
};
 
interface Env { BEACON_KV: KVNamespace; }
 
export default {
  async fetch(req: Request, env: Env, ctx: ExecutionContext) {
    if (req.method !== "POST") return new Response("", { status: 405 });
    const ev = (await req.json()) as BeaconEvent;
    if (!ev?.app || !ev?.kind || !ev?.corrId) {
      return new Response("", { status: 400 });
    }
    const key = `beacon:${ev.app}:${ev.kind}:${ev.corrId}`;
    // 24h TTL; the paired event will delete this key when it arrives
    ctx.waitUntil(env.BEACON_KV.put(key, String(ev.ts), { expirationTtl: 86400 }));
    return new Response("ok");
  },
};

When a paired event arrives, the Beacon is deleted rather than allowed to accumulate. If ad-tap is followed by ad-shown with the same corrId within 6 seconds, both get cleared and the pair is forgotten. The Beacons that remain are the silent-failure candidates.

The Cron does not full-scan KV. It lists by prefix=beacon: hourly and only extracts threshold-breaching ones. Each app produces only a few hundred Beacons per hour, so the Workers CPU budget is more than enough.

Catching missing rewards within 24 hours via receipt reconciliation

The reward-not-granted case is invisible from the client. Cases where the AdMob SDK fails to call onUserEarnedReward — mediation misconfig, SDK version mismatches — have hit me 3 times in 12 years. The most recent one is what cost me the 12% revenue drop I opened this post with.

The catch happens in 3 steps:

1. Front Beacon: emit kind: "rwd-show" the moment RewardedAd.show() is called
2. Dismiss Beacon: emit kind: "rwd-dismiss" from onAdDismissed (correction signal)
3. Reward Beacon: emit kind: "rwd-earned" inside onUserEarnedReward

Then a daily Cron computes per-app "rwd-dismiss arrived but rwd-earned did not" ratios. In my apps the healthy baseline is 0.4 to 1.1% (users who interrupt mid-video, etc). If it exceeds 3% over a rolling 24 hours, Slack pages me. Two such alerts have fired in 3 weeks; one was a mediation-network setting that had been wiped to empty by a UI mis-click.

The Rork (React Native) hook looks like this.

import mobileAds, { RewardedAd, RewardedAdEventType } from "react-native-google-mobile-ads";
 
const ad = RewardedAd.createForAdRequest("ca-app-pub-xxxx/yyyy");
const corrId = () => Math.random().toString(36).slice(2);
 
async function showRewarded(app: string) {
  const id = corrId();
  await beacon(app, "rwd-show", id);
  const unsubLoaded = ad.addAdEventListener(RewardedAdEventType.LOADED, () => ad.show());
  const unsubEarn = ad.addAdEventListener(RewardedAdEventType.EARNED_REWARD, () => {
    beacon(app, "rwd-earned", id);  // ← absence of this is the smoking gun
  });
  ad.addAdEventListener(RewardedAdEventType.CLOSED, () => {
    beacon(app, "rwd-dismiss", id);
    unsubLoaded(); unsubEarn();
  });
  ad.load();
}

Carrying the same corrId through all 3 events is the key. The server can then isolate rwd-show → rwd-dismiss → no rwd-earned, which is exactly the "watched to the end but reward never landed" case.

Catching frozen purchase flows via state-machine stall detection

"Tapped buy, dialog never appeared" is another classic. StoreKit / Billing purchase() can stay in Pending indefinitely without returning an error. Crashlytics never sees it.

A 4-state machine — idle → tapping → presenting → completed / dismissed — works well. If tapping lasts more than 6 seconds the state transitions to stalled and a Beacon is emitted.

type S = "idle" | "tapping" | "presenting" | "completed" | "dismissed" | "stalled";
let s: S = "idle"; let stallTimer: any;
 
async function buyPro(app: string) {
  const id = Math.random().toString(36).slice(2);
  s = "tapping";
  await beacon(app, "purchase-tap", id);
  stallTimer = setTimeout(async () => {
    if (s === "tapping") {           // ← 6s elapsed, never reached presenting
      s = "stalled";
      await beacon(app, "purchase-stall", id);
    }
  }, 6000);
 
  try {
    await Purchases.purchaseProduct("pro_monthly", null, "subs");  // RevenueCat example
    clearTimeout(stallTimer);
    s = "completed";
    await beacon(app, "purchase-ok", id);
  } catch (e: any) {
    clearTimeout(stallTimer);
    s = "dismissed";
    // normal user cancels do not page; purchase-stall is treated separately
  }
}

The threshold here is 0.5% in 24 hours for purchase-stall. The healthy baseline is 0.1 to 0.2%, with most causes being network flakiness and OS dialog non-responsiveness. One alert fired in 3 weeks, which traced back to a StoreKit sandbox outage that leaked subtly into production. Apple's System Status flagged it an hour later.

3 weeks of running it — detection counts and noise ratio

After 3 weeks of parallel operation across all 6 apps, the numbers across all 8 silent-failure categories combined were:

The 64% false-positive rate looks high at first glance, but Slack notifications are reviewed by a human, so it is fine in practice. More importantly, of the 4 real incidents, 1 is estimated to have shaved a week off detection time, which translates to roughly 3% of monthly AdMob revenue preserved. For the first time in 12 years, silent degradation became something I can talk about with numbers.

Cross-app operations — per-app toggles via Remote Config

Running the same Beacon set across 6 apps means one noisy app can flood Slack. I keep beacon_kinds_enabled in Remote Config, per app, so I can:

• Send only rwd-earned from apps trialling a new SDK
• Temporarily disable purchase-stall from apps mid-billing-refactor
• 50% sample all Beacons for the 48 hours after a release

If you combine this with Crashlytics Velocity Alerts, you also have to loosen the Crashlytics thresholds in lockstep, otherwise crashes and silent degradations pile into the same Slack channel without prioritization.

I split them into silent-alerts and crash-alerts channels with distinct colors — blue for silent, red for crash. That visual difference alone has cut my late-night response time by roughly 1.5x, subjectively.

"Silent degradation is worse than crashes" — taking 12 years to learn that

When I started programming in 1997 at 16, I saw code in only two states: it runs or it does not. Run, fix, repeat. Very simple. By the time my wallpaper apps had crossed 50 million cumulative downloads since 2014, I had come to believe the scariest state is the one where the app "appears to work" while quietly degrading.

Both my grandfathers were temple carpenters in Japan, and I watched them work as a child. They would say "this part is still fine" — and then check it again the next day, and the day after that. Keeping working things working costs more than building them in the first place, and far less of the work shows.

The silent-failure layer is not exciting. But it is what protects the revenue of the 6 Rork apps I run, quietly, every day. I hope these notes are useful to anyone else operating multiple apps as a single person.