◇ App Dev/2026-10-26Advanced

Your 5% Rollout Says 99.76% Crash-Free. How Many Users Until You Can Trust It?

At 5% rollout, a crash-free rate rests on few users and the interval is wide. A Wilson-interval script that says proceed, wait or halt, plus simulated numbers on how often you halt a good release.

staged rolloutcrash-freeCrashlytics¹³ Google Play⁴³Wilson intervalrelease decisionsPythonindie development⁴³

✦ Premium Article

The morning after I moved a Google Play staged rollout to 5%, Crashlytics showed "Crash-free users 99.76%". My bar is 99.7%. Above the bar, so on to 25% — and then my hand stopped, because the user count read 420.

The same percentage means different things at different sizes. 99.76% over 420 users may simply mean only one person happened to crash. 99.76% over 20,000 users is close to settled. The dashboard prints both in the same typeface with the same confidence.

As an indie developer I ship in four steps, 5% → 25% → 50% → 100%, and I halt (pull the fraction back) when crash-free users drop below 99.7% or ANR rises above 0.20%. I wrote about this in maintaining my Android wallpaper apps after a release; v2.0.0 was halted twice. I am not changing those lines. What I changed is that I now look at how far the number can be trusted, next to the number.

Below is a small Python script that decides proceed, wait or halt with sample size included, and an unexpected side effect of running it every morning, which I measured by simulating it.

The same 99.76% reads very differently at different sizes

Crash-free is binomial data: k of n users crashed. The observed value can stay the same while the plausible range around it changes a lot with n. Here is the 95% Wilson interval (explained below) for four sizes.

Users	Crashed	Observed crash-free	Plausible true value (95%)	Verdict (target 99.7%)
420	1	99.76%	98.66-99.96%	Wait
2,000	5	99.75%	99.42-99.89%	Wait
5,000	12	99.76%	99.58-99.86%	Wait
20,000	48	99.76%	99.68-99.82%	Wait

All four rows sit above 99.7% as observed. Even at 20,000 users the bottom of the interval, 99.68%, still dips under the target. "It's above the bar, go" holds far later than I expected.

Whatever the first 5% shows is a direction, not a verdict. If you want to use it as evidence, decide in advance what counts as enough evidence.

Decide by where the interval sits relative to the limit

The rule is simple. Look at the interval against the allowed crash rate (0.3% for a 99.7% target).

If even the upper end is within the limit, it is clearly good: proceed.
If even the lower end exceeds the limit, and the count is more than noise, it is clearly bad: halt.
If the interval straddles the limit, you cannot tell yet: wait.

This runs as-is. It uses only the standard library; you type in the user count and crashed-user count that Crashlytics shows.

#!/usr/bin/env python3
"""Decide PROCEED / WAIT / HALT for a staged rollout, sample size included.
Usage: python3 rollout_gate.py --users 420 --crashed 1 --target 99.7
Standard library only. Type in the user count and the crashed-user count
shown in Crashlytics / Play Console."""
import argparse
import math
import sys
 
Z = {90: 1.645, 95: 1.960}  # one-sided 95% = 1.645 / two-sided 95% = 1.960
 
 
def wilson(k: int, n: int, z: float) -> tuple[float, float]:
    """Wilson interval (low, high) for crash rate k/n. Stays inside 0..1 even for small n."""
    if n <= 0:
        raise ValueError("users must be > 0")
    p = k / n
    z2 = z * z
    denom = 1 + z2 / n
    center = (p + z2 / (2 * n)) / denom
    half = z * math.sqrt(p * (1 - p) / n + z2 / (4 * n * n)) / denom
    return max(0.0, center - half), min(1.0, center + half)
 
 
def judge(users: int, crashed: int, target_pct: float, z: float = Z[95],
          min_crashed: int = 3) -> dict:
    allowed = 1 - target_pct / 100          # allowed crash rate (99.7 -> 0.003)
    low, high = wilson(crashed, users, z)
    if low > allowed and crashed >= min_crashed:
        verdict = "HALT"                    # even the lower bound exceeds the limit, and the count is not just noise
    elif high <= allowed:
        verdict = "PROCEED"                 # even the upper bound is within the limit = clearly good
    else:
        verdict = "WAIT"                    # interval straddles the limit = cannot decide yet
    return {
        "users": users, "crashed": crashed,
        "observed_crash_free": 100 * (1 - crashed / users),
        "crash_free_low": 100 * (1 - high), "crash_free_high": 100 * (1 - low),
        "verdict": verdict,
    }
 
 
def users_needed_for_zero_crash(target_pct: float, z: float = Z[95]) -> int:
    """From which user count can we say PROCEED with zero crashes?
    For k=0 the Wilson upper bound is z^2/(n+z^2); solve <= allowed for n."""
    allowed = 1 - target_pct / 100
    return math.ceil(z * z * (1 / allowed - 1))
 
 
def days_to_decide(users: int, needed: int, users_per_day: float) -> int:
    """Days until we reach `needed` users. Pass the recently measured number of
    new users per day who open the new version at this stage."""
    if users >= needed:
        return 0
    if users_per_day <= 0:
        raise ValueError("users_per_day must be > 0")
    return math.ceil((needed - users) / users_per_day)
 
 
def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--users", type=int, required=True)
    ap.add_argument("--crashed", type=int, required=True)
    ap.add_argument("--target", type=float, default=99.7)
    ap.add_argument("--per-day", type=float, default=0,
                    help="measured new users per day on the new version at this stage")
    ap.add_argument("--min-crashed", type=int, default=3,
                    help="below this crash count, answer WAIT instead of HALT")
    ap.add_argument("--conf", type=int, choices=sorted(Z), default=95)
    a = ap.parse_args()
    if a.crashed > a.users:
        print("crashed exceeds users", file=sys.stderr)
        return 2
    r = judge(a.users, a.crashed, a.target, Z[a.conf], a.min_crashed)
    print(f"{r['verdict']}: {r['crashed']}/{r['users']} users crashed "
          f"(observed {r['observed_crash_free']:.2f}% / true value within "
          f"{r['crash_free_low']:.2f}-{r['crash_free_high']:.2f}%)")
    if r["verdict"] == "WAIT":
        need = users_needed_for_zero_crash(a.target, Z[a.conf])
        print(f"  Confirming {a.target}% with zero crashes takes about {need} users")
        if a.per_day > 0:
            print(f"  At the current pace: about {days_to_decide(a.users, need, a.per_day)} more days")
    return {"PROCEED": 0, "WAIT": 10, "HALT": 20}[r["verdict"]]
 
 
if __name__ == "__main__":
    sys.exit(main())

I chose the Wilson interval because the usual normal approximation breaks exactly where we live: small n, zero to a handful of crashes. At k = 0 the normal approximation has zero width, which produces the false confidence of "no crashes, so perfectly safe". Wilson keeps a width at k = 0 and never leaves the 0 to 1 range. The exit codes 0, 10 and 20 let a shell script or CI job branch on the verdict.

Running it:

$ python3 rollout_gate.py --users 420 --crashed 1
WAIT: 1/420 users crashed (observed 99.76% / true value within 98.66-99.96%)
  Confirming 99.7% with zero crashes takes about 1277 users
 
$ python3 rollout_gate.py --users 2000 --crashed 14
HALT: 14/2000 users crashed (observed 99.30% / true value within 98.83-99.58%)

✦

Thank you for reading this far.

Continue Reading

What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.

WHAT YOU'LL LEARN

✦You can now calculate, from your own target, the smallest user count at which a 5% stage crash-free rate justifies moving on

✦You will be able to stop halting good releases on a noisy number by adding a minimum crash count to your halt rule

✦You can estimate how many days remain until a decision is possible, and check the dashboard less often

Secure payment via Stripe · Cancel anytime

✦

Unlock This Article

Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.

Unlock all articles with Membership →

Even zero crashes does not let you proceed right away

What surprised me was how long it takes to proceed even with no crashes at all. For k = 0 the minimum user count depends only on the target (95% interval).

Crash-free target	Minimum users to proceed with zero crashes
99.5%	765
99.7%	1,277
99.9%	3,838

With a 99.7% target, 1,277 users must open the new version and none of them may crash before you can say "clearly good". Tightening the target by 0.2 points to 99.9% triples the required users. A stricter number delays the decision rather than sharpening it.

This is where --per-day matters: the number of new users per day on the new version at this stage. If it is 150 per day at 5%, going from 420 to 1,277 users takes about 6 days. My old default of "24 to 48 hours per stage" cannot reach a statistical decision in the first stage, and now I know that in advance.

So I changed what the 5% stage is for. It is not where I gather evidence to proceed; it is where I stop what is clearly bad. Evidence to proceed comes from the larger populations at 25% and beyond.

Checking every morning halts good releases

Another thing the numbers exposed is how often you look. Imagine re-running the verdict every time 50 more users arrive. Each check has a small false-alarm chance, but across many checks the chance that some check turns red adds up.

To measure it I generated two kinds of release with a random number generator: a good one with a true crash rate of 0.2% (99.8% crash-free) and a bad one at 0.8%. Each is judged every 50 users up to 2,000 users, and a single HALT counts as stopping. 4,000 trials each, fixed seed.

"""Simulate a good release (true crash rate 0.2%) and a bad one (0.8%),
judge every 50 users, and count when and how often HALT fires."""
import random
from rollout_gate import judge, Z
 
def run(true_rate, z, min_crashed, trials=4000, max_users=2000, step=50, seed=7):
    rnd = random.Random(seed)
    halted = 0; first = []
    for _ in range(trials):
        k = 0
        for n in range(1, max_users + 1):
            if rnd.random() < true_rate:
                k += 1
            if n % step == 0 and judge(n, k, 99.7, z, min_crashed)["verdict"] == "HALT":
                halted += 1; first.append(n); break
    first.sort()
    med = first[len(first) // 2] if first else None
    return 100 * halted / trials, med
 
for label, z, mk in [("z=1.645, no count floor", Z[90], 0), ("z=1.96,  no count floor", Z[95], 0),
                     ("z=1.645, >=3 crashes ", Z[90], 3), ("z=1.96,  >=3 crashes ", Z[95], 3)]:
    good, _ = run(0.002, z, mk)
    bad, med = run(0.008, z, mk)
    print(f"{label}: good release halted {good:5.1f}% / bad release caught {bad:5.1f}% (median at user #{med})")

Results:

Rule	Good release halted by mistake	Bad release caught	Median detection point
One-sided 95%, no count floor	16.8%	98.0%	user #200
Two-sided 95%, no count floor	13.9%	96.9%	user #250
One-sided 95%, at least 3 crashes	7.8%	97.7%	user #350
Two-sided 95%, at least 3 crashes	5.0%	96.0%	user #400

The false-halt rate stays much higher than the "95%" label suggests. Without a count floor, roughly one in six healthy releases turned red at some point during the 5% stage. One crash in 50 users is a 2% observed rate, which pushes the lower end above the limit.

Adding "do not halt until at least 3 users have crashed" barely hurt detection (98.0% to 96.0%) while cutting false halts from 16.8% to 5.0%. The cost is a later catch: the median moves from user #200 to user #400, about 200 users. My judgment is that, as a solo developer who notices an overnight halt only the next morning, a false halt hurts more than a 200-user delay, so I use the count floor by default.

Here are the tests:

import math, unittest
from rollout_gate import wilson, judge, users_needed_for_zero_crash, days_to_decide, Z
 
class T(unittest.TestCase):
    def test_zero_crash_boundary(self):
        n = users_needed_for_zero_crash(99.7)
        self.assertEqual(judge(n, 0, 99.7)["verdict"], "PROCEED")
        self.assertEqual(judge(n - 1, 0, 99.7)["verdict"], "WAIT")
    def test_small_sample_never_halts_on_one_crash(self):
        self.assertEqual(judge(50, 1, 99.7)["verdict"], "WAIT")
        self.assertEqual(judge(50, 1, 99.7, min_crashed=0)["verdict"], "HALT")
    def test_clear_regression_halts(self):
        self.assertEqual(judge(2000, 14, 99.7)["verdict"], "HALT")
    def test_interval_stays_in_range(self):
        for n, k in [(1, 0), (1, 1), (10, 10), (100000, 0)]:
            lo, hi = wilson(k, n, Z[95]); self.assertTrue(0 <= lo <= hi <= 1)
    def test_days(self):
        self.assertEqual(days_to_decide(300, 900, 200), 3)
        self.assertEqual(days_to_decide(900, 900, 200), 0)
 
unittest.main(verbosity=1)

The second test came from a wrong expectation. I wrote "one crash in 50 should surely be WAIT" and the version without a count floor returned HALT. The number was twitchier than my intuition. The count floor became a parameter after that.

What the interval cannot see

The intervals above assume users crash independently. Real crashes cluster around particular devices, OS combinations or screens, and heavy users, who crash more, are over-represented early in a rollout.

So I added one rule outside the script: even below 3 crashes, if they share the same exception and the same device model, halt regardless of sample size and look at the cause. In Crashlytics, the affected-user count of new issues for the new version shows that clustering quickly.

The other issue is the denominator. Crash-free is user-based; some Play Console ANR figures are session- or day-based. The same reasoning applies to ANR, but confirm the denominator before typing it into --users. A wrong denominator causes larger errors than the interval math.

Folding it into routine

My routine is three lines.

Run rollout_gate.py twice only: right before raising a stage, and once the next morning. No checks through the day.
HALT means halt. WAIT means read the days from --per-day and set the next check date. PROCEED means raise the stage.
If the 5% stage stays WAIT, do not force a conclusion; define a separate condition for moving to 25% (for example, no new issue is clustered).

Pulling the fraction back can also be done through the Google Play Developer API. The snippet follows the shape of the official docs but I did not run it for this article; check the track and versionCode against your own values before using it.

# Reference only, not executed. Replace YOUR_PACKAGE_NAME and the versionCode.
from googleapiclient.discovery import build
from google.oauth2 import service_account
 
creds = service_account.Credentials.from_service_account_file(
    "service-account.json",
    scopes=["https://www.googleapis.com/auth/androidpublisher"])
svc = build("androidpublisher", "v3", credentials=creds)
 
def set_rollout(package: str, version_code: int, status: str, fraction: float | None = None):
    """status is "inProgress" (staged rollout) or "halted"."""
    edit = svc.edits().insert(packageName=package, body={}).execute()
    release = {"versionCodes": [str(version_code)], "status": status}
    if status == "inProgress" and fraction is not None:
        release["userFraction"] = fraction     # e.g. 0.25
    svc.edits().tracks().update(
        packageName=package, editId=edit["id"], track="production",
        body={"track": "production", "releases": [release]}).execute()
    svc.edits().commit(packageName=package, editId=edit["id"]).execute()
 
# set_rollout("YOUR_PACKAGE_NAME", 49, "halted")

Pitfalls

Comparing the observed crash-free to your bar without looking at n: while the interval straddles the bar, neither "above" nor "below" is a safe claim.
Re-judging often with no count floor: the chance of halting a healthy release climbs. Fix how often and how soon you check.
Tightening only the target: going from 99.7% to 99.9% triples the users you need and delays the call.
Reading zero crashes as perfectly safe: with k = 0 the upper end stays wide at small n.
Letting device or exception clusters drown in the overall number: if a few crashes share a cause, halt anyway.

One thing to do next

Before your next release, look up once how many new users per day your app gets on a new version, and feed it to --per-day. The script will print the days until your target is reachable. If that wait is unacceptable, the number is not the problem; it is a sign to revisit your stage sizes or your target.

I have been pushed around by single-digit crash counts more than once. I hope this calculation gives a steadier footing to anyone standing in front of the morning dashboard with the same hesitation.

For the overall structure of a staged rollout, see a phased release strategy for Rork apps, and for making crashes readable in the first place, turning production crashes into readable reports.

Thank You for Reading

Rork Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.

✦Copy-paste ready implementation code
✦New advanced guides published daily
✦$5/mo or $10 for lifetime access

View Membership →