My generated wallpaper catalog hashed differently on every run — isolating five sources of nondeterminism

Enumeration order — readdir is sorted, opendir is not

"Directory order is not guaranteed" is common advice, and I had internalised it. That belief is part of why I had switched to opendir streaming to save memory. Measuring it told the opposite story.

I built the same file set three times, varying only the order in which files were created, then called readdirSync 50 times on each.

assets-A (creation order)  : 50 runs → 1 distinct  identical to the sorted array
assets-B (shuffled)        : 50 runs → 1 distinct  identical to the sorted array
assets-C (reverse)         : 50 runs → 1 distinct  identical to the sorted array
A, B and C agree on every position (1225/1225)

fs.readdir returns sorted results. libuv sorts the scandir output with strcmp before handing it over. That is a byte comparison, so locale does not enter into it — setting LC_ALL to C, C.UTF-8, en_US.UTF-8 or POSIX produced the same bf263d05356f every time.

opendir skips that sort.

readdir : bf263d05356f
opendir : 87268f8b81ea   matches the sorted array in 1/1225 positions
opendir, repeated 20 times → 1 distinct (stable within one environment)

Stable within one environment was the trap. Run it locally as many times as you like and it looks perfectly well behaved.

It breaks when the filesystem changes. Same file set, three different filesystems:

All three differ. Between the first two, only 2 files out of 1,225 landed in the same position. The ext4 htree orders entries using a hash seed that is per-filesystem, so two ext4 volumes will not agree either.

In other words this was a failure that could never reproduce locally. A fresh clone in CI, a new container layer, a replaced dev machine — the order changes there and the whole artifact reshuffles.

If you want byte-stable output, do not delegate ordering to a library's internals. libuv's sort is an implementation detail, not something the Node documentation promises. Sorting explicitly after you read is the only version that cannot drift.

JSON.stringify reorders integer-like keys

With ordering pinned, the output still refused to match exactly. Next up: object key order.

When a JavaScript object has keys that parse as array indices, those keys ignore insertion order and come out in ascending numeric order.

const o = {};
["100", "9", "10", "1", "0007", "7", "a10", "a9"].forEach((k) => (o[k] = 1));
 
Object.keys(o).join(",");
// → 1,7,9,10,100,0007,a10,a9
JSON.stringify(o);
// → {"1":1,"7":1,"9":1,"10":1,"100":1,"0007":1,"a10":1,"a9":1}

Insertion was 100, 9, 10, 1, ..., yet the numeric-looking keys sort ascending first and everything else follows in insertion order. "0007" with its leading zero, "-1", and "1.5" are not array indices, so they stay on the string side. There is also a ceiling: anything at or above 4294967295 is treated as a plain string.

const t = {};
["4294967294", "4294967295", "4294967296", "5"].forEach((k) => (t[k] = 1));
Object.keys(t).join(",");
// → 5,4294967294,4294967295,4294967296
//   4294967295 and above keep insertion order

What actually hurt here is that this behaviour had been hiding the bug for a long time. Back when every catalog ID was a plain sequence number, insertion order did not matter at all.

Using the real ID set, shuffling insertion order 30 different ways:

With a single non-numeric ID the result is still deterministic, because that one key always lands last. The second one breaks it.

The move to category-prefixed names such as zen_0006 happened more than a year after this script was written. The diff gate started failing the day the naming convention changed, but at the time it read as "the JSON order shifted again," so nobody connected the two.

The fix is small: stop emitting a keyed object, emit an array.

// before: subject to key reordering
items: { "1": {...}, "zen_0006": {...} }
 
// after: the order you chose is the order you get
items: [ { id: "1", ... }, { id: "zen_0006", ... } ]

If the app needs lookup by ID, build a Map at load time. Map iterates in insertion order and never applies the numeric reshuffle.

Keep collation out of the artifact

Now that ordering is mine to decide, the comparator matters. The script used localeCompare, which is environment-dependent.

Sorting the 1,225 filenames with different comparators, measured against the default sort() (UTF-16 code unit comparison):

localeCompare("ja") moves 180 positions, because ICU's Japanese collation orders kanji by reading. Sorting a 15-tag list makes it obvious:

ja  : 🌸桜 10月 2月 abstract Abstract minimal zen Zen ねこ ネコ ミニマル 猫 夜桜 𠮟咤 α波
en  : 🌸桜 10月 2月 abstract Abstract minimal zen Zen α波 ねこ ネコ ミニマル 𠮟咤 夜桜 猫
byte: 10月 2月 Abstract Zen abstract minimal zen α波 ねこ ネコ ミニマル 夜桜 猫 🌸桜 𠮟咤

Under ja, 猫 sorts before 夜桜; under en it sorts after. Two Node builds carrying different ICU versions can disagree on the same data. Pinning artifact order to an ICU version is not a dependency I want to carry for years.

The default sort() has its own edge. It compares UTF-16 code units, so characters outside the BMP disagree with code point order.

UTF-16 code units : 一.jpg  🌸.jpg  𠮟.jpg  﨑.jpg  ～.jpg
code point order  : 一.jpg  﨑.jpg  ～.jpg  🌸.jpg  𠮟.jpg
UTF-8 byte order  : 一.jpg  﨑.jpg  ～.jpg  🌸.jpg  𠮟.jpg

﨑 and ～ live in U+E000–U+FFFF, above the high surrogate range, so the default sort pushes them behind the emoji. If your filenames include variant kanji, that difference is real.

UTF-8 byte order always agrees with code point order, so Buffer.compare became the comparator for the artifact. Human-friendly ordering belongs to the presentation layer, not to a build product — ship a separate display-order field if you need one.

const byBytes = (a, b) =>
  Buffer.compare(Buffer.from(a, "utf8"), Buffer.from(b, "utf8"));
 
items.sort((a, b) => byBytes(a.id, b.id));

Unicode normalization could not be fixed inside the generator

The last one was normalization, and it was the only source I could not resolve in the script itself.

The fixture includes five filenames in NFD, with combining marks — the shape you get when names created on macOS travel to a Linux box unchanged.

NFD: length 11 / 21 UTF-8 bytes
NFC: length 10 / 18 UTF-8 bytes
a === b            → false
a.localeCompare(b) → 0     (identical under collation)
fs.existsSync(NFC path) → false
fs.existsSync(NFD path) → true

They look identical, they compare equal under collation, === still says false, and the NFC path cannot be opened. Normalizing only the catalog keys gives you the worst combination: IDs that match and files that cannot be read.

So what about normalizing the ID while keeping the real filename in a file field? I measured that too.

As long as the raw filename is carried through, the artifact cannot match across platforms. Obvious in hindsight; I genuinely expected normalizing the keys to be enough.

Normalization turned out to be a repository problem, not a build problem. Reject it upstream instead of absorbing it downstream.

// scripts/check-nfc.mjs
import fs from "node:fs";
 
const bad = [];
for (const dir of process.argv.slice(2)) {
  for (const n of fs.readdirSync(dir)) {
    if (n !== n.normalize("NFC")) bad.push(`${dir}/${n}`);
  }
}
 
if (bad.length) {
  console.error(`${bad.length} filenames are not NFC:`);
  for (const b of bad.slice(0, 5)) console.error("  " + JSON.stringify(b));
  process.exit(1);
}
console.log("all filenames are NFC");

$ node scripts/check-nfc.mjs assets/
5 filenames are not NFC:
  "assets/こもれび_0.jpg"
  ...
exit=1

On the macOS side, confirm git config core.precomposeunicode true. It defaults to true, but repositories carrying old configuration sometimes have it off. Once those five filenames were renamed to NFC, readdir order agreed across both platforms.

The deterministic generator

Here is the version with every fix applied. The numbered comments map to the sources above.

// scripts/gen-catalog.mjs
import fsp from "node:fs/promises";
import path from "node:path";
 
const SRC = process.argv[2];
const OUT = process.argv[3];
 
// 1) enumerate with readdir, never with streamed opendir
const names = (await fsp.readdir(SRC)).filter((n) => n.endsWith(".jpg"));
 
// 2) preserve input order (use the Promise.all return value)
const items = await Promise.all(
  names.map(async (name) => {
    const st = await fsp.stat(path.join(SRC, name));
    // 3) normalize the key; check-nfc.mjs guarantees the on-disk name upstream
    const id = name.replace(/\.jpg$/, "").normalize("NFC");
    const cat = id.includes("_") ? id.split("_")[0] : "legacy";
    return {
      id,
      file: `${id}.jpg`,
      category: cat,
      bytes: st.size,
      width: 1290,
      height: 2796,
      // 4) fix the precision of floats before serializing
      aspect: Number((1290 / 2796).toFixed(4)),
    };
  })
);
 
// 5) sort explicitly, with a locale-free comparator
const byBytes = (a, b) =>
  Buffer.compare(Buffer.from(a, "utf8"), Buffer.from(b, "utf8"));
items.sort((a, b) => byBytes(a.id, b.id));
 
const tags = [...new Set(items.map((i) => i.category))].sort(byBytes);
 
// 6) only embed a timestamp when one is supplied from outside
const epoch = process.env.SOURCE_DATE_EPOCH;
const doc = {
  schema: 2,
  generatedAt: epoch ? new Date(Number(epoch) * 1000).toISOString() : null,
  count: items.length,
  tags,
  items,
};
 
// 7) pin key order and end the file with a newline
const stable = (obj) =>
  JSON.stringify(
    obj,
    (_k, v) =>
      v && typeof v === "object" && !Array.isArray(v)
        ? Object.fromEntries(Object.keys(v).sort(byBytes).map((k) => [k, v[k]]))
        : v,
    2
  ) + "\n";
 
await fsp.writeFile(OUT, stable(doc));

A note on the float handling. 1290 / 2796 serializes as 0.4613733905579399. That value is IEEE 754 arithmetic and does not vary by environment, but the more digits you ship, the more room downstream consumers have to disagree when parsing, rounding, and re-emitting. Rounding to 0.4614 keeps the artifact readable and sidesteps -0 serializing as 0 and 1e21 serializing as 1e+21.

SOURCE_DATE_EPOCH is the convention borrowed from reproducible builds. Set it to the commit timestamp in CI and leave it unset locally, and it stops interfering with the diff check.

The ablation

Applying the fixes one at a time, measured over 20 runs in one environment and across two filesystems:

Row three was the dangerous place to stop. Twenty runs locally, one hash, tests green — and still broken in CI. When someone reports a failure that "doesn't reproduce locally," this is the shape I would look for first.

The last two rows change nothing measurable. They are in because libuv's sort is an implementation detail. If a future version changes it, or someone swaps readdir for a glob library, an explicit sort means the artifact does not move. Six months later, "deterministic by accident" and "deterministic on purpose" are not the same thing at all.

Running the finished version 60 times in a row:

v1 (original) : 60 distinct hashes / 142.5 ms per run
v2 (final)    :  1 distinct hash   / 137.7 ms per run

Environment variables made no difference.

I expected the explicit sort to cost something. It measured slightly faster instead, 142.5 ms down to 137.7 ms — dropping the sequential opendir reads outweighed sorting 1,225 entries. At this scale, determinism is effectively free.

Making the gate work

Only now does the diff gate mean anything.

- name: verify filename normalization
  run: node scripts/check-nfc.mjs assets
 
- name: regenerate the catalog and require no diff
  run: |
    node scripts/gen-catalog.mjs assets public/catalog.json
    git diff --exit-code -- public/catalog.json

A passing gate now asserts something real: the committed artifact is reproducible from the current assets. A failing gate means either assets were added without regenerating, or the generation logic changed without updating the output — both failures worth hearing about.

Because it no longer produces noise, committing the generated artifact became a workable practice again, and CI builds got shorter since the file is already there.

There is a second payoff. A deterministic artifact hash makes a sound cache key: unchanged content means an unchanged hash, so no pointless CDN purges and no wasted downloads for users. That design simply does not hold with a nondeterministic generator.

Deduplication across the same catalog is covered in The same wallpaper appeared twice in my catalog — measuring perceptual hash thresholds across 324 images, and a different pairing of generated output with a CI gate is in What Renovate may bump in an Expo app, and what it must never touch.

The order I would work in next time

1. Read the diff. Values changing and positions changing point at different layers
2. Remove the obvious clocks, randomness, and hostnames — then keep going
3. Check async aggregation. A lot of nondeterminism disappears by turning push into return
4. Check the enumeration API. opendir and hand-rolled walkers guarantee nothing
5. Check the output container. Object keys will not stay in the order you inserted them
6. Detach the comparator from locale. Buffer.compare is the safe choice
7. Reject non-NFC filenames upstream. The generator cannot fix them
8. Finally, run the same input N times and count hashes. You are not finished until the count is one

Doing step 8 first would have saved me a day. "The same input produces the same bytes" is a testable property, and because I never tested it, I drifted toward the opposite fix: switching the gate off.

If you commit generated artifacts, determinism is the precondition for that whole design. Operate without it and you end up removing checks one by one, in proportion to how broken it has become.

Thank you for reading. Running your own generator N times and counting the hashes is a good place to start.