One morning every CI check was green. All 42 E2E tests passed. A few hours after I shipped the OTA update, a crash report landed on the payment confirmation screen — a flow the E2E suite tests on every single run.
Green, but not actually passing.
When I traced the logs and found the reason, my chest tightened a little. It wasn't a bug in the test code, nor a fresh defect in the app. Auto-retries had been quietly swallowing a flaky test's red. The checkout test failed roughly one in three runs, CI re-ran it up to three times, and it kept recording the one lucky pass as "green."
These are field notes on noticing that untrustworthy green, measuring flakiness, quarantining it, and leaving only real failures as the thing that gates a release. They're written for indie developers running Detox or Maestro E2E on apps built with Rork or Rork Max — in the order I actually worked through it across my own handful of apps.
How green stops meaning green — retries as an eraser
Most E2E runners ship auto-retry to cope with instability. With Detox on Jest it's jest.retryTimes(2); with Maestro it's a retry loop on the CI side. Both are common.
Retries aren't evil in themselves. On-device and emulator E2E can fail for reasons that have nothing to do with your code — an animation that hasn't settled, a network hiccup. The real problem is recording only whether the test eventually passed.
Retry three times, pass once, mark it green. That recording scheme can't tell these two apart:
| State | Three attempts | Recorded as | What it really means |
|---|---|---|---|
| Stable and healthy | ○ ○ ○ | Green | Fine |
| Flaky | × × ○ | Green | Could break in production any day |
The second row is the trap. It looks green, so nobody notices. But a test failing 67% of the time will, on a real device under worse conditions than your test rig, eventually turn into a genuine red. My checkout flow was exactly that.
The mental shift that helped: treat pass/fail not as a boolean but as a history of attempts. Stop looking only at the final result. Keep the outcome of each attempt across retries, and flakiness turns into a number you can see.
Measuring flake rate — read JUnit XML per attempt
Both Detox (Jest) and Maestro can emit JUnit-format XML, with a <testcase> per test and a <failure> on failure. If you make each retry its own report, you can aggregate the attempt history mechanically.
First, keep retries by appending rather than overwriting. When running Maestro on GitHub Actions, the reliable move is writing each retry to a separate file.
# .github/workflows/e2e.yml (excerpt)
- name: Run Maestro E2E (3 attempts, keep every report)
run: |
for attempt in 1 2 3; do
maestro test .maestro/ \
--format junit \
--output "reports/e2e-attempt-${attempt}.xml" || true
done
# "|| true" lets all attempts run to completion even after a failure;
# the pass/fail decision moves to the aggregation step belowThe || true matters. Fail-fast here would stop the job before the flaky "×" is ever recorded. The point is to pull the verdict out of the runner and hand it to an aggregation script.
Here's a Node script that reads all the XML files and computes a per-test flake rate. Its only dependency is one lightweight XML parser.
// scripts/flake-rate.mjs
import { readdirSync, readFileSync } from "node:fs";
import { XMLParser } from "fast-xml-parser";
const DIR = "reports";
const parser = new XMLParser({ ignoreAttributes: false, attributeNamePrefix: "@_" });
// test name -> { runs: total attempts, fails: failed attempts }
const stats = new Map();
for (const file of readdirSync(DIR).filter((f) => f.endsWith(".xml"))) {
const xml = parser.parse(readFileSync(`${DIR}/${file}`, "utf8"));
const suites = [].concat(xml.testsuites?.testsuite ?? xml.testsuite ?? []);
for (const suite of suites) {
for (const tc of [].concat(suite.testcase ?? [])) {
const name = `${tc["@_classname"] ?? ""} › ${tc["@_name"]}`;
const rec = stats.get(name) ?? { runs: 0, fails: 0 };
rec.runs += 1;
if (tc.failure || tc.error) rec.fails += 1; // any red attempt counts as a failure
stats.set(name, rec);
}
}
}
const rows = [...stats.entries()]
.map(([name, { runs, fails }]) => ({ name, runs, fails, rate: fails / runs }))
.filter((r) => r.fails > 0 && r.fails < r.runs) // drop all-pass and all-fail; keep only flaky
.sort((a, b) => b.rate - a.rate);
for (const r of rows) {
console.log(`${(r.rate * 100).toFixed(0).padStart(3)}% ${r.fails}/${r.runs} ${r.name}`);
}
process.exitCode = 0; // measurement always succeeds; a separate job owns the gateThe condition r.fails > 0 && r.fails < r.runs is the crux. A test that fails on every attempt is a real failure (it should go red elsewhere); one that passes on every attempt is healthy. Only the middle — passes sometimes, fails sometimes — is flaky. Get this wrong and you'll wave a genuine bug through as "just flaky."
The first time I ran this across six apps, the top flake was the checkout flow at 67% (two failures in three attempts). Next was waiting on the push-notification permission dialog, around 40%. Only as numbers did the shape of the swallowed instability finally show.