Exactly 20 Characters, Still Rejected: Putting Character Counting Behind One Boundary

What truncation actually breaks

Silently trimming over-length input is common. value.slice(0, 20) is one line, and that is exactly what I had written.

I measured the damage across 2,000 randomly assembled strings mixing emoji, combining marks, and ordinary characters.

// trunc_probe.mjs — what a UTF-16 slice destroys
const LONE_SURROGATE =
  /[\uD800-\uDBFF](?![\uDC00-\uDFFF])|(?<![\uD800-\uDBFF])[\uDC00-\uDFFF]/;
 
function inspect(source, limit) {
  const cut = source.slice(0, limit);                          // the naive trim
  const roundTrip = Buffer.from(cut, 'utf8').toString('utf8'); // simulates sending it
  return {
    lone: LONE_SURROGATE.test(cut),        // orphaned surrogate left behind
    replaced: roundTrip.includes('�'), // became a replacement character
  };
}

Results across the 2,000 samples:

One result contradicted what I expected. Passing the damaged string through JSON.stringify and JSON.parse produced zero replacement characters. Orphaned surrogates survive as escapes like \uD83D and come back intact. That is precisely why a local test can look clean.

The damage happens the moment the string becomes actual UTF-8 bytes — sending a fetch body, writing a file. Only then do 36.1% of them turn into replacement characters. Green unit tests, corrupted names in production. That asymmetry changed how I write these tests: the round trip now goes through encoding, not just serialization.

Normalization changes the count

The second surprise was that normalizing on the server changes the character count itself.

NFKC expands ㍑ into four characters. One becomes four. Send an exactly-twenty-character name to an API that applies NFKC for search indexing, and the server sees twenty-three and rejects it. The client did nothing wrong.

Halfwidth kana behaves the same way: unchanged under NFC, shorter under NFKC. Which normalization form you choose is not only a question of how you fold variant spellings — it is a precondition of your length validation.

So I fixed a rule: normalization and counting always happen in the same order and the same form, and that order is written in exactly one place. Even when search-time normalization differs, validation always measures the storage form.

One module owns the counting

The fix was to put counting behind a module that both the client and the server import. It lives as a dependency-free TypeScript file readable from React Native and from Cloudflare Workers alike.

// packages/shared/text-length.ts
// The only counting implementation, shared by React Native and the server.
// ⚠️ Never write a .length-based length check anywhere else.
 
/** Normalization applied at storage time. Deliberately separate from search-time NFKC. */
const STORAGE_FORM: 'NFC' = 'NFC';
 
let segmenter: Intl.Segmenter | null = null;
let segmenterChecked = false;
 
function getSegmenter(): Intl.Segmenter | null {
  if (segmenterChecked) return segmenter;
  segmenterChecked = true;
  try {
    // Some runtimes, Hermes among them, may not ship Intl.Segmenter.
    // Check at runtime rather than trusting the type definitions.
    if (typeof Intl !== 'undefined' && typeof (Intl as any).Segmenter === 'function') {
      segmenter = new Intl.Segmenter('en', { granularity: 'grapheme' });
    }
  } catch {
    segmenter = null; // some environments throw on unsupported locales
  }
  return segmenter;
}
 
/** The single entry point for limits, counters, and truncation. */
export function toGraphemes(input: string): string[] {
  const s = input.normalize(STORAGE_FORM);
  const seg = getSegmenter();
  if (seg) return Array.from(seg.segment(s), (x) => x.segment);
  return fallbackSegment(s);
}
 
export function countGraphemes(input: string): number {
  return toGraphemes(input).length;
}
 
/** Truncation that never splits a cluster, so it never creates a broken character. */
export function truncateGraphemes(input: string, limit: number): string {
  if (limit <= 0) return '';
  const g = toGraphemes(input);
  if (g.length <= limit) return g.join('');
  return g.slice(0, limit).join('');
}
 
export type LengthCheck =
  | { ok: true; count: number }
  | { ok: false; count: number; reason: 'too_long' | 'lone_surrogate' };
 
const LONE_SURROGATE =
  /[\uD800-\uDBFF](?![\uDC00-\uDFFF])|(?<![\uD800-\uDBFF])[\uDC00-\uDFFF]/;
 
/** Server-side validation calls this too, so both verdicts always agree. */
export function checkLength(input: string, limit: number): LengthCheck {
  // Storing an already-broken character contaminates everything downstream.
  if (LONE_SURROGATE.test(input)) {
    return { ok: false, count: 0, reason: 'lone_surrogate' };
  }
  const count = countGraphemes(input);
  return count <= limit ? { ok: true, count } : { ok: false, count, reason: 'too_long' };
}

Running truncateGraphemes over 3,000 strings built the same way, with a limit of twenty, produced zero replacement characters and zero over-limit results — against 75.6% breakage from the naive slice under identical conditions.

Putting normalize inside toGraphemes is the important part. Callers get no opportunity to forget it, which keeps the normalization-changes-the-count problem from leaking outside the function.

A fallback for runtimes without Intl.Segmenter

Intl.Segmenter is not guaranteed to exist. The JavaScript engine and build configuration determine how much of Intl ships, so I never assume — I check at runtime.

Falling back to [...s] is not good enough, as the opening table showed: the family emoji counts as 7, and a twenty-character field fills up after three emoji. So the fallback handles the major clustering rules directly.

// text-length.ts (continued) — approximation for runtimes without Intl.Segmenter
const ZWJ = '\u200D';  // ZERO WIDTH JOINER
const VS16 = '\uFE0F'; // VARIATION SELECTOR-16
 
function isRegionalIndicator(cp: number): boolean {
  return cp >= 0x1f1e6 && cp <= 0x1f1ff; // 🇦–🇿, two of them form one flag
}
function isSkinTone(cp: number): boolean {
  return cp >= 0x1f3fb && cp <= 0x1f3ff;
}
function isCombining(cp: number): boolean {
  // Combining marks, Hangul jamo, variation selectors — things that attach to what precedes
  return (
    (cp >= 0x0300 && cp <= 0x036f) ||
    (cp >= 0x1ab0 && cp <= 0x1aff) ||
    (cp >= 0x1dc0 && cp <= 0x1dff) ||
    (cp >= 0x20d0 && cp <= 0x20ff) ||
    (cp >= 0xfe00 && cp <= 0xfe0f) ||
    (cp >= 0x0e31 && cp <= 0x0e3a) ||
    (cp >= 0x11a8 && cp <= 0x11ff) ||
    (cp >= 0xe0100 && cp <= 0xe01ef)
  );
}
 
function fallbackSegment(s: string): string[] {
  const cps = Array.from(s);
  const out: string[] = [];
  let i = 0;
  while (i < cps.length) {
    let cluster = cps[i];
    i++;
    // Two regional indicators make a single flag grapheme
    if (isRegionalIndicator(cluster.codePointAt(0)!) &&
        i < cps.length && isRegionalIndicator(cps[i].codePointAt(0)!)) {
      cluster += cps[i];
      i++;
      out.push(cluster);
      continue;
    }
    // Absorb trailing combining marks, skin tones, VS16, and ZWJ sequences
    while (i < cps.length) {
      const next = cps[i];
      const cp = next.codePointAt(0)!;
      if (next === VS16 || isSkinTone(cp) || isCombining(cp)) {
        cluster += next;
        i++;
        continue;
      }
      if (next === ZWJ && i + 1 < cps.length) {
        cluster += next + cps[i + 1];
        i += 2;
        continue;
      }
      break;
    }
    out.push(cluster);
  }
  return out;
}

This approximation is not complete. Unicode's grapheme cluster rules are considerably more detailed, and some Hangul and Indic sequences will still be miscounted. But covering emoji and combining marks removes essentially every case where the limit disagrees with what a person sees.

I chose this fallback on the basis of how it degrades rather than how correct it is. With Intl.Segmenter you get the real answer; without it you get a less precise answer that fails in the same direction. Falling back to [...s] fails in a different direction entirely, which is what makes it hard to reason about.

Speed is not the deciding factor here

My initial assumption was that Intl.Segmenter was too heavy to call on every keystroke. Measuring it showed that assumption had nothing behind it.

As a ratio, Intl.Segmenter is roughly 55 times slower than .length. That number alone is enough to scare anyone off.

In absolute terms it is 11.65 µs per call. Counting a forty-grapheme string once measured 28.7 µs, which is 0.172% of a 16.7 ms frame budget at 60fps. Recomputing on every keystroke does not drop a single frame.

Judging by ratio gives one answer; judging by absolute cost against a budget gives the opposite. Whenever performance is the stated reason for accepting incorrect behavior, that reasoning deserves an actual measurement. After seeing these numbers I moved to calling countGraphemes on every change without hesitation.

One caveat: constructing an Intl.Segmenter is not free, which is why the code above creates one at module scope and reuses it. Calling new on every keystroke would invalidate this measurement entirely.

Agreeing with the storage layer

Sharing a function between client and server accomplishes nothing if the database counts in yet another unit. I fixed this as a checklist.

1. Size the column as grapheme limit × worst-case bytes. The family emoji is 25 bytes, so a twenty-grapheme display name wants at least 500 bytes. Writing the same number in the column definition as in the UI limit was the easiest trap to fall into.
2. Route all API validation through checkLength. Framework-provided max-length validators almost always measure UTF-16 units or bytes, so they are unsuitable for character limits.
3. Drive the on-screen counter from the same function, so "20 / 20 followed by a rejection" becomes structurally impossible.
4. Reject orphaned surrogates at the entrance. Rows that already contain a broken character cannot be repaired afterward; blocking them on the way in is the only remedy.
5. Audit existing rows. In my case, records from the era of slice-based trimming were still there. Running checkLength across the table isolates exactly the rows that return lone_surrogate.

I also dropped the maxLength prop on TextInput. It operates on UTF-16 code units, so it will always disagree with a grapheme-based counter. Truncation happens inside onChangeText instead.

// NameField.tsx — no maxLength, only the shared function
import { useCallback, useState } from 'react';
import { Text, TextInput, View } from 'react-native';
import { countGraphemes, truncateGraphemes } from '@shared/text-length';
 
const LIMIT = 20;
 
export function NameField({ initial = '' }: { initial?: string }) {
  const [value, setValue] = useState(initial);
  const count = countGraphemes(value);
 
  const onChangeText = useCallback((next: string) => {
    // Trim by grapheme instead of relying on maxLength; never put a broken character in state.
    setValue(next.length > value.length ? truncateGraphemes(next, LIMIT) : next);
  }, [value.length]);
 
  return (
    <View>
      <TextInput
        value={value}
        onChangeText={onChangeText}
        // Intentionally no maxLength={LIMIT} — it counts UTF-16 units
        accessibilityLabel="Display name"
      />
      <Text>{count} / {LIMIT}</Text>
    </View>
  );
}

One caution for IME users. In environments where onChangeText fires mid-composition, truncating an unconfirmed string can break the conversion. If you have seen characters disappear or the cursor jump during input, Fix Disappearing Characters and Jumping Cursors in Rork TextInputs (3 Patterns) covers the related symptoms; you may need to restrict truncation to confirmed text.

The order I would do it in

None of this had to land at once. In order of payoff:

1. Create the shared module (half a day). countGraphemes and truncateGraphemes alone are enough to eliminate the client/server disagreement.
2. Swap the server-side validation (one to two hours). Doing this first means existing clients start saving successfully and the review reports stop.
3. Remove maxLength from the input (thirty minutes), and point the counter at the shared function.
4. Add the orphaned-surrogate check at the entrance (one hour). The audit of existing rows can wait.
5. Write a round-trip test that goes through encoding (half a day). Zero failures via JSON.stringify versus 36.1% via UTF-8 is exactly what makes this test worth having.

The Intl.Segmenter fallback can come last. Until you actually hit a runtime without it, the runtime existence check is enough.

Decide how you count, in one place. Obvious once written down — and obvious enough that nobody had treated it as a design decision. Other multilingual checks share the same shape; for quantity expressions I covered it in Designing Quantity Strings with CLDR Plural Categories and Intl.PluralRules.

It started with one short review, but working out why my own code was contradicting my own code left me lighter than I expected. I hope it saves you the same detour.