libiconv clib

Character encoding conversion for PascalAI. GNU libiconv converts text between 100+ legacy and modern encodings — UTF-8, GBK, Shift-JIS, CP1252, KOI8-R, Latin-1, Big5, and more.

ppm install libiconv

What is libiconv?

GNU libiconv is the reference implementation for character set conversion. It covers 100+ encodings spanning every major script: Unicode (UTF-8/16/32), Western European, Cyrillic, Chinese, Japanese, Korean, Arabic, Hebrew, Greek, Turkish, and Thai.

On Linux, iconv is built directly into glibc and requires no additional library. The well-known iconv command-line tool is a thin wrapper over the same API this package exposes.

Why encoding conversion is needed

Legacy systems — Windows desktops, old databases, email servers, mainframes — store text in non-UTF-8 encodings. A file saved in GBK on a Chinese Windows machine, or a CSV export using CP1252 from an old ERP, will display as mojibake unless converted to UTF-8 before processing. Convert to UTF-8 first; work in UTF-8 everywhere.

Supported Encodings

Region / FamilyEncodings
UnicodeUTF-8, UTF-16LE, UTF-16BE, UTF-32LE, UTF-32BE, UTF-7
Western EuropeanISO-8859-1..15, CP1252 (Windows Latin-1), MacRoman
Central EuropeanISO-8859-2, CP1250, CP852
CyrillicISO-8859-5, KOI8-R, KOI8-U, CP1251, CP866
CJK — ChineseGBK (CP936), GB18030, Big5 (CP950), EUC-TW
CJK — JapaneseEUC-JP, Shift-JIS (CP932), ISO-2022-JP
CJK — KoreanEUC-KR (CP949), ISO-2022-KR
ArabicISO-8859-6, CP1256
HebrewISO-8859-8, CP1255
GreekISO-8859-7, CP1253
TurkishISO-8859-9, CP1254
ThaiTIS-620, CP874
Transliteration modes

Append //TRANSLIT to the target encoding to substitute similar characters when an exact mapping does not exist (e.g. pass 'ASCII//TRANSLIT' as ToEncoding to approximate Unicode characters with their ASCII look-alikes). Append //IGNORE to silently drop bytes that cannot be converted instead of raising an error. Both suffixes work with any encoding name: 'CP1252//TRANSLIT', 'GBK//IGNORE', etc.

Types

TypeDescription
TIconvCDConversion descriptor handle returned by OpenCD. Represents an open conversion context between two fixed encodings. Must be closed with CloseCD when no longer needed.

API Reference

Open / Close

FunctionDescription
OpenCD(ToEncoding, FromEncoding)Open a conversion descriptor. ToEncoding and FromEncoding are encoding name strings such as 'UTF-8', 'GBK', or 'ISO-8859-1'. Append //TRANSLIT or //IGNORE to ToEncoding for fallback behavior. Returns a TIconvCD handle.
CloseCD(CD)Release a conversion descriptor and free all associated resources.

Convert

FunctionDescription
Convert(CD, Input)Convert Input using an already-open TIconvCD. Efficient for repeated conversions between the same encoding pair — the descriptor is reused without re-opening.
ConvertString(Input, FromEnc, ToEnc)One-shot conversion without managing a descriptor. Opens, converts, and closes internally. Convenient for infrequent conversions where performance is not critical.

Common Conversions (convenience)

FunctionDescription
ToUTF8(Input, FromEncoding)Convert from any encoding to UTF-8.
FromUTF8(Input, ToEncoding)Convert from UTF-8 to any encoding. Supports //TRANSLIT and //IGNORE suffixes.
GBKToUTF8(Input)Convert GBK (Chinese GB2312/GBK, Windows code page 936) to UTF-8.
UTF8ToGBK(Input)Convert UTF-8 to GBK.
SJISToUTF8(Input)Convert Shift-JIS (Japanese Windows, CP932) to UTF-8.
CP1252ToUTF8(Input)Convert CP1252 (Windows Western European / Latin-1 superset) to UTF-8.
KOI8RToUTF8(Input)Convert KOI8-R (legacy Russian encoding) to UTF-8.
Latin1ToUTF8(Input)Convert ISO-8859-1 (Latin-1) to UTF-8.
Big5ToUTF8(Input)Convert Big5 (Traditional Chinese, Taiwan/Hong Kong) to UTF-8.

Detection

FunctionDescription
DetectEncoding(Data)Heuristically detect the encoding of a raw byte sequence. Returns an encoding name string, or 'UTF-8' when detection is inconclusive. Not guaranteed to be correct — use only when the encoding is truly unknown.
IsValidUTF8(Data)Check whether a byte sequence is well-formed UTF-8. Returns Boolean. Useful for routing: if the input is already valid UTF-8, skip conversion entirely.

File

ProcedureDescription
ConvertFile(InPath, OutPath, FromEnc, ToEnc)Read the file at InPath, convert its encoding from FromEnc to ToEnc, and write the result to OutPath. Handles large files without loading the entire content into memory.

Info

FunctionDescription
ListEncodings()Return an array of string containing all encoding names supported by the installed libiconv.
LibIconvVersion()Return the libiconv library version string (e.g. '1.17').

Code Example

uses libiconv;
var
  CD: TIconvCD;
  GBKText, UTF8Text: string;
begin
  { Read a GBK-encoded file (common in China) }
  GBKText := ReadFile('/data/chinese_data.txt');  { raw bytes }

  { Convert to UTF-8 for processing }
  UTF8Text := GBKToUTF8(GBKText);
  WriteLn(UTF8Text);

  { Or use a CD for repeated conversions (more efficient) }
  CD := OpenCD('UTF-8', 'GBK');
  UTF8Text := Convert(CD, GBKText);
  CloseCD(CD);

  { Convert UTF-8 back to CP1252 for a Windows app }
  WriteLn(FromUTF8('Héllo Wörld', 'CP1252//TRANSLIT'));
end.

Common Use Cases

Install

ppm install libiconv

Built into glibc on Linux — no install needed. brew install libiconv on macOS.

Package Info

Version1.0.0
Typeclib
CategoryEncoding
Encodings100+

API

  • OpenCD / CloseCD
  • Convert(CD, Input)
  • ConvertString
  • ToUTF8 / FromUTF8
  • GBKToUTF8 / UTF8ToGBK
  • SJISToUTF8
  • CP1252ToUTF8
  • KOI8RToUTF8
  • Latin1ToUTF8
  • Big5ToUTF8
  • DetectEncoding
  • IsValidUTF8
  • ConvertFile
  • ListEncodings
  • LibIconvVersion
Related packages

For full Unicode normalization, locale-aware collation, and script transliteration see libicu. For regex with Unicode category support see pcre2.