What is utf8proc?
utf8proc is a small, dependency-free C library that implements the core operations needed to correctly handle UTF-8 text in modern software. Rather than treating strings as raw bytes, it understands Unicode: normalization forms, combining characters, grapheme clusters, bidirectional classes, display widths, and the full Unicode general category system.
Despite its small footprint, utf8proc is trusted at scale. It powers the Julia language runtime, is embedded in PostgreSQL for Unicode-aware string handling, used by Git for path normalization, and found in numerous text editors for correct cursor positioning and text layout. When you need correct Unicode handling without pulling in the full ICU stack, utf8proc is the standard choice.
Why Unicode normalization matters
The same character can have multiple byte representations.
The letter
Normalize before comparing or storing. Use
The letter
é can be encoded as U+00E9 (a single precomposed
codepoint) or as U+0065 + U+0301 (the letter e
followed by a combining acute accent). Both render identically in any font, but they are
not byte-equal and will not match in a naive string comparison.Normalize before comparing or storing. Use
ToNFC for web content
and database storage; use CaseFold on top of normalization for case-insensitive
search. Skipping normalization causes silent bugs in search, deduplication, URL routing, and
user identity systems.
Normalization forms
| Form | Full name | Description |
|---|---|---|
| NFC | Canonical Decomposition + Canonical Composition | Most compact form. Precomposed characters are preferred. Default for web, databases, and most storage formats. |
| NFD | Canonical Decomposition | Characters split into base letter + combining diacritic marks. Useful for stripping accents or processing diacritics separately. |
| NFKC | Compatibility Decomposition + Canonical Composition | Like NFC but also flattens compatibility characters: ligatures (fi, ffl), half-width katakana, circled digits, superscripts. Use for identifier comparison. |
| NFKD | Compatibility Decomposition | Fully decomposed including compatibility equivalences. Used in Unicode identifier analysis and security-sensitive normalization. |
Grapheme clusters
What a user sees as one character may be many codepoints.
The rainbow flag emoji 🏳️🌈 is a single grapheme cluster but is composed of multiple Unicode codepoints joined by a Zero Width Joiner. Skin tone modifiers, combining diacritics, and regional indicator pairs (flag emojis) all produce grapheme clusters that span two or more codepoints.
The rainbow flag emoji 🏳️🌈 is a single grapheme cluster but is composed of multiple Unicode codepoints joined by a Zero Width Joiner. Skin tone modifiers, combining diacritics, and regional indicator pairs (flag emojis) all produce grapheme clusters that span two or more codepoints.
Length(s) in most languages returns the codepoint count, not the
grapheme count. For correct cursor positioning, text truncation,
and character counting as perceived by a human reader, use
GraphemeCount and GetGraphemes.
Types
| Type | Description |
|---|---|
| TU8NormForm | Enum of normalization forms: nfNFC=0, nfNFD=1, nfNFKC=2, nfNFKD=3. Passed to Normalize(). |
| TU8Category | Unicode general category enum with 30 values: ucLetterUppercase, ucLetterLowercase, ucMarkNonspacing (combining diacritics), ucNumberDecimal, ucPunctDash, ucSymbolMath, ucSeparatorSpace, ucControl, ucFormat, ucUnassigned, and more. |
| TU8GraphemeBreak | Grapheme break property enum used internally to determine cluster boundaries: gbCR, gbLF, gbControl, gbExtend, gbZWJ, gbSpacingMark, gbPrepend, gbRegionalInd (flag emoji components), gbOther. |
| TU8CharInfo | Record holding all properties for a single Unicode codepoint: Codepoint: Integer, Category: TU8Category, CombiningClass: Integer (0 = not combining), BidiClass: Integer, Width: Integer (0/1/2 display columns), IsLower, IsUpper, IsAlpha, IsDigit, IsSpace, IsEmoji: Boolean. |
API Reference
Normalization
function Normalize(S: string; Form: TU8NormForm): string
Normalize a UTF-8 string to the given form. Use
nfNFC for most purposes.function ToNFC(S: string): string
Shorthand for
Normalize(S, nfNFC). Canonical Decomposition followed by Canonical Composition. Preferred for web content and database storage.function ToNFD(S: string): string
Shorthand for
Normalize(S, nfNFD). Decomposes all precomposed characters into base + combining marks. Useful for accent stripping (apply NFD then filter out ucMarkNonspacing codepoints).function ToNFKC(S: string): string
Shorthand for
Normalize(S, nfNFKC). Compatibility composition: flattens ligatures, half-width characters, and other compatibility variants. Use for identifier normalization and search tokenization.Case folding
function ToLower(S: string): string
Unicode-aware lowercase. Handles Turkish dotless i (ı), German sharp s (ß), Greek sigma forms, and all other language-specific casing rules.
function ToUpper(S: string): string
Unicode-aware uppercase conversion.
function ToTitle(S: string): string
Titlecase conversion: uppercases the first letter of each word following Unicode word-boundary rules.
function CaseFold(S: string): string
Case-fold for case-insensitive comparison. More thorough than
ToLower: maps German ß to ss, Greek and Coptic characters to their Latin equivalents, etc. Use this for hashing and equality comparison, not for display.function CompareCI(A, B: string): Integer
Case-insensitive comparison. Returns −1 if A < B, 0 if equal, 1 if A > B. Equivalent to comparing
CaseFold(A) with CaseFold(B).Codepoint iteration
function CodepointCount(S: string): Integer
Return the number of Unicode codepoints in S. For ASCII text this equals the byte length; for multibyte characters it is less. Does not count grapheme clusters.
function GetCodepoints(S: string): array of Integer
Decode the UTF-8 string into an array of integer codepoint values (UTF-32). Useful for codepoint-level manipulation before re-encoding.
function FromCodepoints(Codepoints: array of Integer): string
Encode an array of integer codepoints back into a UTF-8 string.
function EncodeCodepoint(Codepoint: Integer): string
Encode a single codepoint to its 1–4 byte UTF-8 representation.
Grapheme clusters
function GraphemeCount(S: string): Integer
Return the number of user-perceived characters (grapheme clusters) in S. This is the correct value to display as the string length in a UI. Emoji sequences and characters with combining marks count as 1 each.
function GetGraphemes(S: string): array of string
Split S into its grapheme clusters, each returned as a UTF-8 substring. Iterate this array for correct cursor movement, text selection, and truncation.
Character properties
function GetCharInfo(Codepoint: Integer): TU8CharInfo
Return all Unicode properties for a single codepoint in one call: category, combining class, bidi class, display width, and boolean flags (IsAlpha, IsDigit, IsEmoji, etc.).
function GetCategory(Codepoint: Integer): TU8Category
Return the Unicode general category for a codepoint. Useful for filtering: skip punctuation, detect digits, find combining marks.
function GetWidth(Codepoint: Integer): Integer
Return the display column width for a codepoint: 0 for null and non-spacing combining marks, 1 for most characters, 2 for CJK ideographs and fullwidth forms.
function StringWidth(S: string): Integer
Return the total display column width of a string. Sums
GetWidth for each codepoint. Use this for terminal alignment, padding, and truncation to a fixed column count.Validation
function IsValidUTF8(S: string): Boolean
Return True if S contains only well-formed UTF-8 byte sequences. Use at API boundaries to validate untrusted input before processing.
function SanitizeUTF8(S: string): string
Remove or replace invalid UTF-8 byte sequences with the Unicode replacement character (U+FFFD). Safe to apply to arbitrary binary data.
Info
function UnicodeVersion: string
Return the Unicode standard version supported by this build of utf8proc, e.g.
'15.1.0'.function Utf8ProcVersion: string
Return the utf8proc library version string, e.g.
'2.9.0'.Code Example
Normalization, case folding, grapheme iteration, and display width in one program:
uses utf8proc;
var
S, Norm : string;
Graphemes: array of string;
I : Integer;
begin
S := 'Héllo Wörld'; { mixed precomposed and decomposed }
Norm := ToNFC(S); { normalize to NFC for storage }
WriteLn('NFC: ', Norm);
WriteLn('Lowercase: ', ToLower('ÑOÑO')); { Spanish }
WriteLn('CaseFold: ', CaseFold('Straße')); { German: strasse }
{ Grapheme iteration for correct cursor/truncation }
Graphemes := GetGraphemes('Hello 🌍!');
for I := 0 to High(Graphemes) do
WriteLn(' [', I, '] "', Graphemes[I], '" width=',
StringWidth(Graphemes[I]));
{ Display width for terminal alignment }
WriteLn('Width of "你好": ', StringWidth('你好')); { 4 — CJK = 2 columns each }
end.Category lookup example
Using GetCharInfo and GetCodepoints to classify characters in a string:
uses utf8proc;
var
Points: array of Integer;
Info : TU8CharInfo;
CP, I : Integer;
begin
Points := GetCodepoints('café 😀 combîned');
for I := 0 to High(Points) do
begin
CP := Points[I];
Info := GetCharInfo(CP);
if Info.IsEmoji then
WriteLn('U+', IntToHex(CP, 4), ' — emoji, width=', Info.Width)
else if Info.Category = ucMarkNonspacing then
WriteLn('U+', IntToHex(CP, 4), ' — combining mark (class ',
Info.CombiningClass, ')')
else if Info.IsAlpha then
WriteLn('U+', IntToHex(CP, 4), ' — letter')
else if Info.IsSpace then
WriteLn('U+', IntToHex(CP, 4), ' — whitespace');
end;
end.