Exactly 20 Characters, Still Rejected: Putting Character Counting Behind One Boundary

What truncation actually breaks

Silently trimming over-length input is common. value.slice(0, 20) is one line, and that is exactly what I had written.

I measured the damage across 2,000 randomly assembled strings mixing emoji, combining marks, and ordinary characters.

// trunc_probe.mjs — what a UTF-16 slice destroys
const LONE_SURROGATE =
  /[\uD800-\uDBFF](?![\uDC00-\uDFFF])|(?<![\uD800-\uDBFF])[\uDC00-\uDFFF]/;
 
function inspect(source, limit) {
  const cut = source.slice(0, limit);                          // the naive trim
  const roundTrip = Buffer.from(cut, 'utf8').toString('utf8'); // simulates sending it
  return {
    lone: LONE_SURROGATE.test(cut),        // orphaned surrogate left behind
    replaced: roundTrip.includes('�'), // became a replacement character
  };
}

Results across the 2,000 samples:

One result contradicted what I expected. Passing the damaged string through JSON.stringify and JSON.parse produced zero replacement characters. Orphaned surrogates survive as escapes like \uD83D and come back intact. That is precisely why a local test can look clean.

The damage happens the moment the string becomes actual UTF-8 bytes — sending a fetch body, writing a file. Only then do 36.1% of them turn into replacement characters. Green unit tests, corrupted names in production. That asymmetry changed how I write these tests: the round trip now goes through encoding, not just serialization.

Normalization changes the count

The second surprise was that normalizing on the server changes the character count itself.

NFKC expands ㍑ into four characters. One becomes four. Send an exactly-twenty-character name to an API that applies NFKC for search indexing, and the server sees twenty-three and rejects it. The client did nothing wrong.

Halfwidth kana behaves the same way: unchanged under NFC, shorter under NFKC. Which normalization form you choose is not only a question of how you fold variant spellings — it is a precondition of your length validation.

So I fixed a rule: normalization and counting always happen in the same order and the same form, and that order is written in exactly one place. Even when search-time normalization differs, validation always measures the storage form.

One module owns the counting

The fix was to put counting behind a module that both the client and the server import. It lives as a dependency-free TypeScript file readable from React Native and from Cloudflare Workers alike.

// packages/shared/text-length.ts
// The only counting implementation, shared by React Native and the server.
// ⚠️ Never write a .length-based length check anywhere else.
 
/** Normalization applied at storage time. Deliberately separate from search-time NFKC. */
const STORAGE_FORM: 'NFC' = 'NFC';
 
let segmenter: Intl.Segmenter | null = null;
let segmenterChecked = false;
 
function getSegmenter(): Intl.Segmenter | null {
  if (segmenterChecked) return segmenter;
  segmenterChecked = true;
  try {
    // Some runtimes, Hermes among them, may not ship Intl.Segmenter.
    // Check at runtime rather than trusting the type definitions.
    if (typeof Intl !== 'undefined' && typeof (Intl as any).Segmenter === 'function') {
      segmenter = new Intl.Segmenter('en', { granularity: 'grapheme' });
    }
  } catch {
    segmenter = null; // some environments throw on unsupported locales
  }
  return segmenter;
}
 
/** The single entry point for limits, counters, and truncation. */
export function toGraphemes(input: string): string[] {
  const s = input.normalize(STORAGE_FORM);
  const seg = getSegmenter();
  if (seg) return Array.from(seg.segment(s), (x) => x.segment);
  return fallbackSegment(s);
}
 
export function countGraphemes(input: string): number {
  return toGraphemes(input).length;
}
 
/** Truncation that never splits a cluster, so it never creates a broken character. */
export function truncateGraphemes(input: string, limit: number): string {
  if (limit <= 0) return '';
  const g = toGraphemes(input);
  if (g.length <= limit) return g.join('');
  return g.slice(0, limit).join('');
}
 
export type LengthCheck =
  | { ok: true; count: number }
  | { ok: false; count: number; reason: 'too_long' | 'lone_surrogate' };
 
const LONE_SURROGATE =
  /[\uD800-\uDBFF](?![\uDC00-\uDFFF])|(?<![\uD800-\uDBFF])[\uDC00-\uDFFF]/;
 
/** Server-side validation calls this too, so both verdicts always agree. */
export function checkLength(input: string, limit: number): LengthCheck {
  // Storing an already-broken character contaminates everything downstream.
  if (LONE_SURROGATE.test(input)) {
    return { ok: false, count: 0, reason: 'lone_surrogate' };
  }
  const count = countGraphemes(input);
  return count <= limit ? { ok: true, count } : { ok: false, count, reason: 'too_long' };
}

Running truncateGraphemes over 3,000 strings built the same way, with a limit of twenty, produced zero replacement characters and zero over-limit results — against 75.6% breakage from the naive slice under identical conditions.

Putting normalize inside toGraphemes is the important part. Callers get no opportunity to forget it, which keeps the normalization-changes-the-count problem from leaking outside the function.

A fallback for runtimes without Intl.Segmenter

Intl.Segmenter is not guaranteed to exist. The JavaScript engine and build configuration determine how much of Intl ships, so I never assume — I check at runtime.

Falling back to [...s] is not good enough, as the opening table showed: the family emoji counts as 7, and a twenty-character field fills up after three emoji. So the fallback handles the major clustering rules directly.

// text-length.ts (continued) — approximation for runtimes without Intl.Segmenter
const ZWJ = '\u200D';  // ZERO WIDTH JOINER
const VS16 = '\uFE0F'; // VARIATION SELECTOR-16
 
function isRegionalIndicator(cp: number): boolean {
  return cp >= 0x1f1e6 && cp <= 0x1f1ff; // 🇦–🇿, two of them form one flag
}
function isSkinTone(cp: number): boolean {
  return cp >= 0x1f3fb && cp <= 0x1f3ff;
}
function isCombining(cp: number): boolean {
  // Combining marks, Hangul jamo, variation selectors — things that attach to what precedes
  return (
    (cp >= 0x0300 && cp <= 0x036f) ||
    (cp >= 0x1ab0 && cp <= 0x1aff) ||
    (cp >= 0x1dc0 && cp <= 0x1dff) ||
    (cp >= 0x20d0 && cp <= 0x20ff) ||
    (cp >= 0x3099 && cp <= 0x309a) || // combining kana voiced marks (missing in my first version)
    (cp >= 0xfe00 && cp <= 0xfe0f) ||
    (cp >= 0x0e31 && cp <= 0x0e3a) ||
    (cp >= 0x0e47 && cp <= 0x0e4e) ||
    (cp >= 0x093a && cp <= 0x094f) || // Devanagari vowel signs and virama
    (cp >= 0x0951 && cp <= 0x0957) ||
    (cp >= 0x1160 && cp <= 0x11ff) || // Hangul vowel and trailing jamo (first version started at 0x11a8)
    (cp >= 0xe0100 && cp <= 0xe01ef)
  );
}
 
function fallbackSegment(s: string): string[] {
  const cps = Array.from(s);
  const out: string[] = [];
  let i = 0;
  while (i < cps.length) {
    let cluster = cps[i];
    i++;
    // Two regional indicators make a single flag grapheme
    if (isRegionalIndicator(cluster.codePointAt(0)!) &&
        i < cps.length && isRegionalIndicator(cps[i].codePointAt(0)!)) {
      cluster += cps[i];
      i++;
      out.push(cluster);
      continue;
    }
    // Absorb trailing combining marks, skin tones, VS16, and ZWJ sequences
    while (i < cps.length) {
      const next = cps[i];
      const cp = next.codePointAt(0)!;
      if (next === VS16 || isSkinTone(cp) || isCombining(cp)) {
        cluster += next;
        i++;
        continue;
      }
      if (next === ZWJ && i + 1 < cps.length) {
        cluster += next + cps[i + 1];
        i += 2;
        continue;
      }
      break;
    }
    out.push(cluster);
  }
  return out;
}

This approximation is not complete. Rather than leave "not complete" as a phrase, I wanted a number for it, so I treated Intl.Segmenter as ground truth and scored the fallback against it.

Scoring the fallback against the real thing

My first version — the isCombining above without the two lines marked "missing in my first version", and without the Thai tone mark and Devanagari ranges — agreed with Intl.Segmenter on 9 of the 11 rows in the opening table. The two misses were が written as a base kana plus a combining voiced mark, and 각 written as three separate jamo. The combining dakuten at U+3099 and the Hangul vowel jamo at U+1161 were simply absent from my range list.

I noticed my own fallback was losing to my own table only after I ran the comparison. A single pass at the time I built that table would have caught it, and I'll admit my shoulders dropped a little.

// fallback_probe.mjs — score the fallback against Intl.Segmenter
const seg = new Intl.Segmenter('ja', { granularity: 'grapheme' });
const real = (s) => Array.from(seg.segment(s)).length;
 
let agree = 0, over = 0, total = 0;
for (const s of randomStrings(3000)) {   // 5–34 items mixing emoji, combining marks, plain text
  const a = real(s);
  const b = fallbackSegment(s).length;
  if (a === b) agree++;
  over += Math.abs(a - b);
  total += a;
}
console.log(`agree ${agree}/3000  overcount ${over}/${total}`);

Here are the results over 3,000 strings built from the same material, with [...s] included for scale.

Even the first version is in a different league from [...s], but overcounting by nearly a tenth means a twenty-character field would shortchange people by roughly two characters. On this material, the two missing lines — combining dakuten and Hangul vowel jamo — account for the entire gap. I added the Thai tone mark and Devanagari ranges in the same pass, but they were not in the test material, so their effect would need a separate measurement.

That 100% comes with a condition: it holds for this material. The Devanagari conjunct क्ष, which Intl.Segmenter counts as 1, still comes out as 2 from the corrected fallback, and नमस्ते comes out 3 versus 4. The virama rule that links consonants cannot be expressed as a range list, and I have left that gap open. If your app serves Indic scripts, I may be missing something, but I would not rely on this approximation — treat a runtime with Intl.Segmenter as a requirement instead.

I chose this fallback on the basis of how it degrades rather than how correct it is. With Intl.Segmenter you get the real answer; without it you get a less precise answer that fails in the same direction. What separates this from falling back to [...s] is that I can now say, in numbers, what the degradation looks like.

A fallback only earns the name once it ships with its agreement rate against the real implementation. An unmeasured approximation is not a fallback — it is a second source of truth.

Speed is not the deciding factor here

My initial assumption was that Intl.Segmenter was too heavy to call on every keystroke. Measuring it showed that assumption had nothing behind it.

As a ratio, Intl.Segmenter is roughly 55 times slower than .length. That number alone is enough to scare anyone off.

In absolute terms it is 11.65 µs per call. Counting a forty-grapheme string once measured 28.7 µs, which is 0.172% of a 16.7 ms frame budget at 60fps. Recomputing on every keystroke does not drop a single frame.

Judging by ratio gives one answer; judging by absolute cost against a budget gives the opposite. Whenever performance is the stated reason for accepting incorrect behavior, that reasoning deserves an actual measurement. After seeing these numbers I moved to calling countGraphemes on every change without hesitation.

One caveat: constructing an Intl.Segmenter is not free, which is why the code above creates one at module scope and reuses it. Calling new on every keystroke would invalidate this measurement entirely.

Agreeing with the storage layer

Sharing a function between client and server accomplishes nothing if the database counts in yet another unit. I fixed this as a checklist.

1. Size the column as grapheme limit × worst-case bytes. The family emoji is 25 bytes, so a twenty-grapheme display name wants at least 500 bytes. Writing the same number in the column definition as in the UI limit was the easiest trap to fall into.
2. Route all API validation through checkLength. Framework-provided max-length validators almost always measure UTF-16 units or bytes, so they are unsuitable for character limits.
3. Drive the on-screen counter from the same function, so "20 / 20 followed by a rejection" becomes structurally impossible.
4. Reject orphaned surrogates at the entrance. Rows that already contain a broken character cannot be repaired afterward; blocking them on the way in is the only remedy.
5. Audit existing rows. In my case, records from the era of slice-based trimming were still there. Running checkLength across the table isolates exactly the rows that return lone_surrogate.

I also dropped the maxLength prop on TextInput. It operates on UTF-16 code units, so it will always disagree with a grapheme-based counter. Truncation happens inside onChangeText instead.

// NameField.tsx — no maxLength, only the shared function
import { useCallback, useState } from 'react';
import { Text, TextInput, View } from 'react-native';
import { countGraphemes, truncateGraphemes } from '@shared/text-length';
 
const LIMIT = 20;
 
export function NameField({ initial = '' }: { initial?: string }) {
  const [value, setValue] = useState(initial);
  const count = countGraphemes(value);
 
  const onChangeText = useCallback((next: string) => {
    // Trim by grapheme instead of relying on maxLength; never put a broken character in state.
    setValue(next.length > value.length ? truncateGraphemes(next, LIMIT) : next);
  }, [value.length]);
 
  return (
    <View>
      <TextInput
        value={value}
        onChangeText={onChangeText}
        // Intentionally no maxLength={LIMIT} — it counts UTF-16 units
        accessibilityLabel="Display name"
      />
      <Text>{count} / {LIMIT}</Text>
    </View>
  );
}

One caution for IME users. In environments where onChangeText fires mid-composition, truncating an unconfirmed string can break the conversion. If you have seen characters disappear or the cursor jump during input, Fix Disappearing Characters and Jumping Cursors in Rork TextInputs (3 Patterns) covers the related symptoms; you may need to restrict truncation to confirmed text.

The order I would do it in

None of this had to land at once. In order of payoff:

1. Create the shared module (half a day). countGraphemes and truncateGraphemes alone are enough to eliminate the client/server disagreement.
2. Swap the server-side validation (one to two hours). Doing this first means existing clients start saving successfully and the review reports stop.
3. Remove maxLength from the input (thirty minutes), and point the counter at the shared function.
4. Add the orphaned-surrogate check at the entrance (one hour). The audit of existing rows can wait.
5. Write a round-trip test that goes through encoding (half a day). Zero failures via JSON.stringify versus 36.1% via UTF-8 is exactly what makes this test worth having.

The Intl.Segmenter fallback can come last. Until you actually hit a runtime without it, the runtime existence check is enough. When you do add one, score it against the real segmenter once, as in the section above, before you let it ship.

Decide how you count, in one place. Obvious once written down — and obvious enough that nobody had treated it as a design decision. Other multilingual checks share the same shape; for quantity expressions I covered it in Designing Quantity Strings with CLDR Plural Categories and Intl.PluralRules.

It started with one short review, but working out why my own code was contradicting my own code left me lighter than I expected. I hope it saves you the same detour.