All operations

Unicode normalize

Rewrite text into one of the four standard forms so identical-looking strings are stored identically.

What is loaded above

The input holds fullwidth Latin letters, the fi ligature, a circled five, and the Roman numeral twelve as a single code point. NFKC flattens it to plain ASCII. Switch to NFC and the line comes back untouched: none of those characters are canonically equivalent to their ASCII counterparts, only compatible.

Composed and decomposed

The letter é can be stored two ways: as one code point, U+00E9, or as a plain e followed by U+0301 COMBINING ACUTE ACCENT. Both draw the same glyph. NFC gives you the single code point, NFD the pair.

String equality fails, hashes differ, and a unique index takes the two forms as two rows. HFS+ stored every filename decomposed, so a name created on a Mac stopped matching the same name typed on Linux. Git carries a core.precomposeUnicode setting for that reason, and turns it on in every repository it creates on a Mac.

The compatibility forms

NFKC and NFKD go further, folding away distinctions that are formatting rather than identity. Superscripts, ligatures, fullwidth forms, circled and parenthesized letters, Roman numerals and the mathematical alphabets collapse to ordinary letters and digits.

That fold is lossy on purpose and this operation has no reverse. Once the text says 2, nothing records that it arrived as a superscript.

Getting it wrong at the login screen

Two strings a person reads as identical should compare as identical to whatever decides who you are. Usernames, email local parts and domains want normalizing before comparison, in the same form on both sides of the check.

Doing it at the wrong moment opens the hole. Spotify hit it in June 2013 and published the details. Its function for deriving a canonical username was not idempotent: ᴮᴵᴳᴮᴵᴿᴰ, written in modifier capital letters, came back as BIGBIRD after one pass and bigbird after two. Register the first form, ask for a password reset, follow the link, and the extra pass puts the new password on the real bigbird’s account. Account creation stayed switched off until it was fixed.

Normalize on the way in, store the normalized form, enforce uniqueness on that. Folding only at comparison time lets two rows exist that are one user.

What it will not do

NFKC leaves Cyrillic alone. Write раypal with a Cyrillic а and р and all four forms return it unchanged: they are different letters. Compatibility folding cleans up typographic variants and offers nothing against a homograph domain.

Inspect catches the mixed-script case this operation cannot. Unicode code points gives the identity of every character before and after a fold. Invisible characters handles the ones normalization leaves in place.