Every Experiment Kept Landing on the Same Devices: Hash Choice and Salt Position, Measured Across a Million IDs

The hash that passed uniformity broke the moment a second experiment appeared

The mistake was looking at one experiment at a time.

Assign experiment A and experiment B at 10% each over the same devices and count how many land in both. Under independence that is 1.00% of all devices, or 10.00% of the devices already in A.

function overlap(fn, saltA, saltB) {
  let a = 0, both = 0;
  for (let i = 0; i < N; i++) {
    const inA = fn(saltA + ids[i]) % 100 < 10;
    if (inA) {
      a++;
      if (fn(saltB + ids[i]) % 100 < 10) both++;
    }
  }
  return { aPct: (a / N) * 100, conditional: (both / a) * 100 };
}

Forgetting the salt entirely gives 100%, no surprise. Row three is the surprise. The polynomial hash that sailed through uniformity at 103.0 correlates at 65.20% across two experiments — matching the "six in ten" I had seen in production almost exactly.

Adding a third experiment widened the gap. Three experiments at 10% each should put 0.100% of devices in all three.

Seventy-two times over. Without meaning to, I had manufactured a permanent test cohort — a group absorbing every change at once. Shortened onboarding, altered paywall copy, and a third unrelated feature, all aimed at the same people. Beyond contaminating the readings, it concentrated every disruption on a small set of users, which bothered me more than the statistics did.

The culprit was the concatenation order, not the hash

This is where I sat still the longest.

A 31-based polynomial hash returns, for a string c₀c₁…c_{n-1}:

h(s) = Σ c_i · 31^(n-1-i)  (mod 2^32)

Build the key as salt + ':' + id and the ID length is constant, because it is always a UUID. The salt therefore always gets multiplied by the same 31^37, and the ID part is simply added on top:

h(saltA + ':' + id) = 31^37 · h(saltA + ':') + h(id)
h(saltB + ':' + id) = 31^37 · h(saltB + ':') + h(id)

Subtract, and h(id) cancels. What remains is a constant that does not depend on the device at all. Collecting the differences over 20,000 devices:

const diffs = new Set();
for (let i = 0; i < 20000; i++) {
  const d = (javaHash('exp_paywall_v2:' + ids[i]) - javaHash('exp_onboard_v3:' + ids[i])) >>> 0;
  diffs.add(d);
}
console.log(diffs.size); // → 1

One distinct value across all 20,000 devices: 3716953708. Run the same loop with FNV-1a and you get 19,998 distinct differences.

Experiment A and experiment B were the same number line, shifted sideways by a constant. Where that shift happens to land decides whether the overlap comes out at 65% or 90%. Nothing about randomness quality was at fault; a hash that mixes characters purely by addition is simply the wrong shape for this job.

Moving the salt to the end fixes it — conditionally

Once the structure is visible, the fix suggests itself: write id + ':' + salt, so the ID gets multiplied by a power that depends on the salt's length, which differs per experiment. Measured:

Different lengths settle at 10.08%. Equal lengths drop to 0.00% — perfectly exclusive. Not one device in experiment A appears in experiment B.

Same algebra, of course. Equal salt lengths mean equal multipliers on the ID, which puts us back at a constant difference. This time the shift happens to carry the 10% window cleanly outside itself.

That is where I abandoned the trailing-salt fix. Keys like exp_paywall_v2 and exp_onboard_v3 end up the same length all the time. A design that depends on a rule saying "never let two experiment keys have equal length" is a design my future self will violate within a season. Naming conventions for flags are worth having — I wrote up mine in Making Feature Flags Survive Production: Kill Switches and Gradual Rollouts in Rork Max — but the accidents a naming rule should prevent and the accidents a hash function should prevent are better kept separate.

A final mix to clean up the low bits

FNV-1a with a leading salt came in at a healthy 10.32%. Move the salt to the end, though, and it drifts to 12.92%. Taking the bucket with % 100 uses the low bits, and FNV-1a's avalanche in the low bits is shallow after the last multiply, so strings that differ only at the tail retain a trace of correlation.

One round of MurmurHash3's finalizer (fmix32) closes it:

function fmix32(h) {
  h ^= h >>> 16;
  h = Math.imul(h, 0x85ebca6b);
  h ^= h >>> 13;
  h = Math.imul(h, 0xc2b2ae35);
  h ^= h >>> 16;
  return h >>> 0;
}
const bucketHash = (s) => fmix32(fnv1a(s));

Re-measured over 500,000 IDs:

With the extra round, either concatenation order lands near 10%. The triple-enrolment rate comes out at exactly 0.100%, and a 34/33/33 three-way split measures 34.00% / 33.04% / 32.96%.

Cost per call is 0.496µs, against 0.510µs without the finalizer. Five lines that remove the need to reason about concatenation order forever — that was not a hard call.

Keeping devices in place during a ramp-up

There is one more operation every experiment eventually reaches: watch it at 10%, then widen to 20%.

If a device already seeing the new experience gets pushed back out, the person perceives a feature disappearing. On a subscription path, that costs you people who were one tap from purchasing.

Three ways to write it:

Only the threshold form is monotone in the exposure. Bucket numbers stay pinned per device and the boundary is the only thing that moves, so nobody who is inside ever falls out.

Versioned salts turn out to be the dangerous one precisely because they look meaningful. You notice that bumping to v2 reshuffles everyone right after you have bumped to v2. The release-side mechanics of staged rollout are covered separately in A Phased Release Strategy for Rork Apps.

I settled on 10,000 buckets. One-percent steps cover almost everything, but 0.1% granularity earns its keep when walking a kill switch back, where a coarse step makes the retreat clumsy.

The implementation I settled on

This is the assignment layer shared across six apps. The only dependencies are expo-secure-store and @react-native-async-storage/async-storage; the hash is local.

// src/experiments/bucketing.ts
const FNV_OFFSET = 0x811c9dc5;
const FNV_PRIME = 0x01000193;
export const BUCKET_RESOLUTION = 10_000; // 0.01% granularity
 
function fnv1a32(input: string): number {
  let h = FNV_OFFSET;
  for (let i = 0; i < input.length; i++) {
    h ^= input.charCodeAt(i);
    h = Math.imul(h, FNV_PRIME); // a plain * overflows past 2^53
  }
  return h >>> 0;
}
 
function fmix32(h: number): number {
  h ^= h >>> 16;
  h = Math.imul(h, 0x85ebca6b);
  h ^= h >>> 13;
  h = Math.imul(h, 0xc2b2ae35);
  h ^= h >>> 16;
  return h >>> 0;
}
 
/** Bucket 0–9999 from an experiment key and a unit ID. Synchronous, deterministic. */
export function bucketOf(experimentKey: string, unitId: string): number {
  if (!experimentKey || !unitId) {
    // An empty string would drop every device into the same bucket. Fail loudly.
    throw new Error(`bucketOf: empty key or unitId (key="${experimentKey}")`);
  }
  return fmix32(fnv1a32(`${experimentKey}:${unitId}`)) % BUCKET_RESOLUTION;
}
 
export type Variant<T extends string> = { name: T; weight: number };
 
/**
 * Weighted variant selection. Weights need not sum to 10000;
 * whatever is left over falls to the first entry, i.e. the control group.
 */
export function pickVariant<T extends string>(
  experimentKey: string,
  unitId: string,
  variants: readonly Variant<T>[],
): T {
  if (variants.length === 0) throw new Error(`pickVariant: no variants for "${experimentKey}"`);
  const total = variants.reduce((sum, v) => sum + v.weight, 0);
  if (total > BUCKET_RESOLUTION) {
    throw new Error(`pickVariant: weights exceed ${BUCKET_RESOLUTION} in "${experimentKey}"`);
  }
 
  const bucket = bucketOf(experimentKey, unitId);
  let cursor = 0;
  for (const v of variants) {
    cursor += v.weight;
    if (bucket < cursor) return v.name;
  }
  return variants[0].name; // outside the allocation = control
}
 
/** Ramp-ups. ratioBps runs 0–10000 (10000 = 100%). Raising it never evicts anyone. */
export function isInRollout(flagKey: string, unitId: string, ratioBps: number): boolean {
  const clamped = Math.max(0, Math.min(BUCKET_RESOLUTION, Math.floor(ratioBps)));
  return bucketOf(flagKey, unitId) < clamped;
}

The unit ID side writes to both the Keychain and AsyncStorage, so either one can restore the value:

// src/experiments/unit-id.ts
import * as SecureStore from 'expo-secure-store';
import AsyncStorage from '@react-native-async-storage/async-storage';
import * as Crypto from 'expo-crypto';
 
const KEY = 'exp.unit_id.v1';
let cached: string | null = null;
 
export function getUnitIdSync(): string | null {
  return cached; // the synchronous path used during launch
}
 
export async function loadUnitId(): Promise<string> {
  if (cached) return cached;
 
  // 1) Keychain / Keystore — survives reinstall on iOS
  try {
    const secure = await SecureStore.getItemAsync(KEY);
    if (secure) return (cached = await mirror(secure));
  } catch (e) {
    // Device lock state can make this unreadable. Do not crash here.
    console.warn('[exp] secure read failed', e);
  }
 
  // 2) AsyncStorage — the fallback when the Keychain is unavailable
  try {
    const plain = await AsyncStorage.getItem(KEY);
    if (plain) return (cached = await mirror(plain));
  } catch (e) {
    console.warn('[exp] async read failed', e);
  }
 
  // 3) Mint a new one
  const fresh = Crypto.randomUUID();
  return (cached = await mirror(fresh));
}
 
async function mirror(id: string): Promise<string> {
  await Promise.allSettled([
    SecureStore.setItemAsync(KEY, id),
    AsyncStorage.setItem(KEY, id),
  ]);
  return id;
}

Callers await loadUnitId() once at startup and evaluate synchronously from then on. Where the first paint has to be correct, such as a paywall, an unloaded ID returns the control variant. Loading finishes in a dozen or so milliseconds, so in practice this only affects the very first frame of a first launch.

// src/experiments/use-variant.ts
import { useMemo } from 'react';
import { getUnitIdSync } from './unit-id';
import { pickVariant, type Variant } from './bucketing';
 
export function useVariant<T extends string>(
  experimentKey: string,
  variants: readonly Variant<T>[],
): T {
  const unitId = getUnitIdSync();
  return useMemo(() => {
    if (!unitId) return variants[0].name; // control while the ID is still loading
    return pickVariant(experimentKey, unitId, variants);
  }, [experimentKey, unitId, variants]);
}

Analytics events carry the bucket number alongside the variant name, so the delivered exposure can be re-derived downstream rather than trusted. That paid off again when aligning with the setup in Building a Rork A/B Testing Platform with GrowthBook and PostHog from Scratch.

Why I skipped SHA-256, and it was not about speed

On the numbers, SHA-256 held up well on independence (P(B|A) 9.85%). It costs 2.781µs per call against FNV-1a's 0.510µs — 5.4 times more — but evaluating thirty flags at launch still totals only 83.4µs. Speed was never the deciding factor.

The decider was elsewhere. React Native has no synchronous SHA-256. expo-crypto's digestStringAsync returns a promise, which puts an await between launch and the variant you wanted resolved before the first paint. A native module gets you a synchronous version, but whether to add one purely for experiment assignment is a separate decision.

Nothing here needs cryptographic strength. Nobody is trying to guess their assignment. The requirement was mixing, and only mixing. Reach for the heavier tool on a misread requirement and you give up the one that actually mattered.

Do not over-trust a uniformity test

One more thing surfaced while measuring.

Supposing the unit ID were a sequential counter rather than a UUID, I ran the same uniformity check:

Chi-squared of 0.8. With 99 degrees of freedom, that sits in the lower rejection region, not the upper one. It is too even. Feed a counter through a linear hash and buckets get filled in orderly rotation, producing a distribution with none of the jitter real randomness has.

Uniformity tests are usually framed as a way to catch skew. For assignment design, "suspiciously smooth" turned out to be just as telling. Seeing that number is what made me stop treating uniformity as the pass condition on its own. Uniformity, independence, and stickiness now get measured together, every time.

The four rules that came out of it

Four lines went into the README of the shared layer:

1. Build keys as experimentKey:unitId and hash only with fnv1a32 followed by fmix32 — no other hash enters the codebase
2. Express exposure as a threshold over 10,000 buckets — ramp by raising ratioBps, and never version the salt
3. Whenever a new experiment lands, check its co-enrolment rate against the existing ones once — a gap over two points from the expectation means a design bug
4. Send both the variant name and the bucket number with analytics events — exposure accidents should be catchable downstream

The third would not have occurred to me otherwise. Tests around assignment layers gravitate toward "is it deterministic," and the relationship between experiments falls straight through the gap.

If you want to know whether the same trap is sitting in your codebase, take your assignment function and measure the co-enrolment rate for two experiment keys. At 10% and 10%, the conditional probability should land near 10%. That one check reproduces everything here in your own environment.

For what it is worth, I spent most of that day suspecting the ID generator. Only after lining the numbers up did the actual suspect come into view. I hope it saves you the day.