ASCII vs Unicode vs UTF-16: Character Encodings Compared

Every developer eventually meets a string that looks like Café instead of Café, a CSV file whose first column header starts with an invisible character, or an emoji that makes length return 2. These bugs are not random. They come from a small set of ideas about how text becomes bytes, and once you understand those ideas, you can predict and fix almost every one of them. This guide compares ASCII, the legacy code pages, Unicode, UTF-8, UTF-16 and UTF-32 side by side, using byte counts measured by real code, so you can choose the right encoding for files, databases, APIs and email, and diagnose mojibake when something goes wrong.

It is written for working developers: backend engineers who store and ship text, frontend engineers who wonder why "😀".length is 2, and anyone who has inherited a database full of question marks. You do not need to memorise the Unicode tables. You need a clear mental model, a few reliable tools and a short checklist. By the end you will know the difference between a character set and an encoding, how UTF-8 and UTF-16 lay out the same text, why the web settled on UTF-8, and how to migrate old data without destroying it.

Character sets versus encodings

Two words get mixed up constantly, and the confusion causes real bugs, so separate them first.

A character set (or character repertoire) is a list of characters, each with a number. The number is called a code point. In Unicode, the capital letter A is code point 65, written U+0041, and the euro sign is U+20AC. A character set says nothing about bytes. It only answers the question "which number is this character?"

An encoding is the rule for turning those numbers into bytes and back. The same code point can become different bytes under different encodings. The euro sign, U+20AC, is the three bytes e2 82 ac in UTF-8, the two bytes 20 ac in UTF-16 big-endian, and the single byte 80 in Windows-1252, which is a different character set altogether. The Wikipedia article on character encoding describes the general idea, and the W3C introduction to character encodings explains it in a web context.

A useful way to hold this in your head is a three-layer model:

1. Characters are what a human sees: letters, symbols, emoji.
2. Code points are the numbers a character set assigns to them.
3. Bytes are what an encoding produces and what files and networks actually carry.

Older standards blurred layers two and three, because ASCII maps a character to a number that is also the byte. That is why many programmers of a certain age say "ASCII code" when they mean "byte". Unicode deliberately separates the layers, because one number space has to serve many encodings.

Why the distinction matters in practice

Whenever you read or write text, you cross from bytes to characters or back. Bytes alone cannot tell you which encoding they used. The byte e9 is é in Latin-1, part of a multi-byte sequence in UTF-8, and a fragment of a Japanese character in Shift_JIS. Software needs to be told, or must guess. Nearly all encoding bugs reduce to one of three situations:

• The writer used one encoding and the reader assumed another.
• A conversion step silently replaced characters it could not represent.
• A string was split or measured in units that are not characters.

Keep those three failure modes in mind as we go. Every section below maps back to one of them.

A short history: from 7 bits to a universal set

ASCII

The American Standard Code for Information Interchange defines 128 characters using 7 bits, the values 0 to 127. The first 32 values (0 to 31), plus 127, are control codes such as line feed (10), carriage return (13), tab (9) and escape (27). The remaining 95 are printable: the space, digits, upper and lower case English letters and a set of punctuation. The letter A is 65, a is 97, and the digit 0 is 48.

ASCII's design has useful quirks. Upper and lower case letters differ by exactly one bit, 32, so a case conversion is a single bit flip. Digits map to their values by subtracting 48. These properties made ASCII convenient for early machines, and they still show up in modern code. You can read more in the ASCII article on Wikipedia.

ASCII has one glaring limit: it only covers English. There is no é, no ñ, no €, no Greek, Cyrillic, Arabic, Hebrew, Hindi or Chinese. Because computers store data in 8-bit bytes, the unused eighth bit looked like free real estate.

The age of code pages

Vendors and standards bodies filled the upper 128 values (128 to 255) with whatever their market needed. The result was dozens of incompatible 8-bit encodings called code pages. Each keeps ASCII in the lower half and puts different characters in the upper half.

• ISO-8859-1 (Latin-1) covers Western European languages. Its 256 values map one-to-one onto the first 256 Unicode code points, which makes it uniquely easy to convert.
• Windows-1252 is Microsoft's variant. It resembles Latin-1 but replaces the range 0x80 to 0x9F, which Latin-1 reserves for rarely used control characters, with useful punctuation such as the euro sign, curly quotes and the en dash. Many documents labelled Latin-1 are actually Windows-1252, and web browsers treat the label iso-8859-1 as Windows-1252 for exactly that reason.
• ISO-8859-2, -5, -7, -9 and others cover Central European, Cyrillic, Greek, Turkish and further scripts. KOI8-R was widely used for Russian.
• Shift_JIS, EUC-JP, GBK, Big5 and EUC-KR are multi-byte encodings for East Asian languages, where one byte is nowhere near enough. They use a mix of single-byte ASCII and double-byte sequences.

The core problem with code pages is that the file itself does not say which one it uses. A Russian document in KOI8-R opened as Windows-1252 turns into nonsense. You also cannot mix scripts: a single 8-bit code page cannot hold French and Greek in the same document.

Unicode

Unicode set out to fix this by giving every character in every writing system its own number, in one universal character set. The code space runs from U+0000 to U+10FFFF, which is 1,114,112 possible code points, divided into 17 planes of 65,536 code points each. Plane 0, the Basic Multilingual Plane (BMP), holds the characters for most living languages. The other planes (the supplementary planes) hold historic scripts, rarer CJK ideographs, many symbols and most emoji. Code points are written as U+ followed by at least four hex digits.

The first 128 Unicode code points are identical to ASCII, and the first 256 are identical to Latin-1. This was a deliberate compatibility decision that makes conversion from legacy data easier than it might have been. The Unicode article on Wikipedia gives a good overview of how the standard is organised and maintained.

Unicode is a character set, not an encoding. It defines encoding forms that turn code points into code units and then into bytes: UTF-8, UTF-16 and UTF-32. The rest of this guide is about those three.

A note on UCS-2

Early Unicode designers assumed 65,536 code points would be enough, so they proposed a fixed 16-bit encoding, now called UCS-2. Systems built in that era, including Windows NT, Java and JavaScript, adopted 16-bit characters. When Unicode outgrew 16 bits, those systems could not switch to a four-byte width without breaking everything, so UTF-16 was devised to extend UCS-2 backwards compatibly using surrogate pairs. This history explains why UTF-16 appears inside so many platforms today, and why its quirks leak into string length and indexing.

How the three Unicode encodings work

Before looking at details, here is a comparison of the three forms and the legacy 8-bit ones.

Property	ASCII	Latin-1 / Windows-1252	UTF-8	UTF-16	UTF-32
Code unit size	7 bits (stored in 8)	8 bits	8 bits	16 bits	32 bits
Bytes per character	1	1	1 to 4	2 or 4	4
Characters covered	128	256	All of Unicode	All of Unicode	All of Unicode
ASCII-compatible	Yes	Yes	Yes	No	No
Byte order issue	No	No	No	Yes	Yes
Fixed width	Yes	Yes	No	No	Yes

UTF-8: variable width, ASCII-safe

UTF-8 encodes each code point in one to four bytes. The leading bits of the first byte tell a decoder how long the sequence is, and every continuation byte begins with the bits 10.

Code point range	Bytes	Byte pattern
U+0000 to U+007F	1	0xxxxxxx
U+0080 to U+07FF	2	110xxxxx 10xxxxxx
U+0800 to U+FFFF	3	1110xxxx 10xxxxxx 10xxxxxx
U+10000 to U+10FFFF	4	11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

The x positions carry the bits of the code point. Here is a worked example with the euro sign, U+20AC. In binary, 0x20AC is 0010 0000 1010 1100. It falls in the three-byte range, so we split those 16 bits into groups of 4, 6 and 6 bits: 0010, 000010, 101100. Insert them into the pattern 1110xxxx 10xxxxxx 10xxxxxx and you get 11100010 10000010 10101100, which is hex e2 82 ac. That matches the byte sequence shown earlier.

This design gives UTF-8 several excellent properties:

• ASCII is unchanged. Any valid ASCII text is already valid UTF-8, byte for byte. Tools that only understand ASCII keep working on the ASCII parts of UTF-8 text.
• No byte order problem. The code unit is a single byte, so there is no endianness.
• Self-synchronising. You can land in the middle of a stream, skip continuation bytes (those that match 10xxxxxx) and find the start of the next character. A damaged byte affects one character, not everything after it.
• No embedded zero bytes except for the actual U+0000 character, so C-style null-terminated string functions keep working.
• Sorting by bytes matches sorting by code point. Comparing UTF-8 byte sequences gives the same order as comparing code points.

The cost is that characters outside ASCII take more bytes, and you cannot jump to the nth character without scanning. UTF-8 is defined in RFC 3629, which also limits it to code points up to U+10FFFF and forbids encoding the surrogate range.

Another important rule: UTF-8 requires the shortest form. The ASCII slash, U+002F, must be the single byte 2f. The overlong two-byte sequence c0 af decodes to the same number mathematically, but a conforming decoder must reject it. Overlong forms were used in real attacks to smuggle path separators past security filters, which is why strict decoders matter.

UTF-16: 16-bit units and surrogate pairs

UTF-16 stores each code point in one or two 16-bit code units. Code points in the BMP (U+0000 to U+FFFF, excluding the surrogate range) are stored directly as one unit, so the euro sign is the single unit 0x20AC. Code points above U+FFFF need two units, called a surrogate pair.

The range U+D800 to U+DFFF is permanently reserved: it holds no characters, and exists only so that UTF-16 can use it for pairs. The algorithm is short:

1. Subtract 0x10000 from the code point. This leaves a 20-bit number.
2. The top 10 bits, added to 0xD800, give the high surrogate (range D800 to DBFF).
3. The bottom 10 bits, added to 0xDC00, give the low surrogate (range DC00 to DFFF).

For the grinning face emoji, U+1F600: subtract 0x10000 to get 0xF600. The top ten bits are 0x3D, and 0xD800 plus 0x3D is 0xD83D. The bottom ten bits are 0x200, and 0xDC00 plus 0x200 is 0xDE00. The pair is D83D DE00, which is exactly what JavaScript reports for "😀".charCodeAt(0) and charCodeAt(1), as we will see in the code section. The UTF-16 article on Wikipedia covers the algorithm and its history, and RFC 2781 specifies it for internet use.

Because a high surrogate can only be followed by a low surrogate, and the two ranges never overlap with BMP characters, a decoder can always tell which kind of unit it is looking at. That is the same self-synchronising idea, applied to 16-bit units.

A UTF-16 string is well-formed only when every high surrogate is followed by a low surrogate and no low surrogate stands alone. JavaScript, Java and Windows strings are really arbitrary sequences of 16-bit units, so they can contain unpaired surrogates that are not valid Unicode text. That is the source of several nasty bugs, covered in the pitfalls section.

Byte order and the BOM

A 16-bit unit is two bytes, and there are two orders in which to store them. Big-endian (BE) stores the most significant byte first, and little-endian (LE) stores the least significant byte first. The capital letter A, U+0041, is 00 41 in UTF-16BE and 41 00 in UTF-16LE.

When a file does not carry that information elsewhere, it may begin with a byte order mark (BOM), the character U+FEFF. Written in BE it is bytes fe ff, and in LE ff fe. A reader looks at the first two bytes, learns the order, and strips the mark. The same trick works for UTF-32, where the little-endian BOM is ff fe 00 00.

The BOM is also available in UTF-8, as the three bytes ef bb bf. There is no byte order to signal, so in UTF-8 it only acts as a signature saying "this file is UTF-8." This is controversial. Some Windows tools add it and rely on it, while many Unix tools, parsers and shell scripts treat the bytes as stray content. We return to this in the pitfalls and best practices sections. The byte order mark article on Wikipedia lists the signatures for each encoding.

If you name an encoding explicitly as UTF-16LE or UTF-16BE, no BOM is expected or written. If you say plain UTF-16, the BOM is used to detect the order, and many libraries default to a BOM plus the platform's native order when writing.

UTF-32: fixed width, expensive

UTF-32 stores every code point as a single 32-bit value. The euro sign is 0000 20AC. Its big advantage is simplicity, since the nth code point is at byte offset 4n, so random access is trivial. Its disadvantage is cost: every character takes four bytes, even plain ASCII, so text is four times larger than in ASCII-compatible encodings. It also has byte order variants and a BOM.

UTF-32 is rarely used for storage or transmission. It appears occasionally as an internal representation, for example in some C libraries and in languages that choose code point indexing, but even those often optimise the storage.

Same text, different bytes: measured results

Theory is easier to trust when you can run it. This Python script encodes four sample strings in several encodings and prints the byte counts. It shows n/a where the encoding cannot represent the text.

samples = {
    "ASCII": "Hello, world!",
    "Accented Latin": "Café déjà vu, señor",
    "CJK": "你好，世界",
    "Emoji": "😀👍🏽",
}
encodings = ["ascii", "latin-1", "utf-8", "utf-16-le", "utf-32-le"]
for name, text in samples.items():
    sizes = []
    for enc in encodings:
        try:
            sizes.append(f"{enc}={len(text.encode(enc))}")
        except UnicodeEncodeError:
            sizes.append(f"{enc}=n/a")
    print(f"{name:15} chars={len(text):2}", " ".join(sizes))

Output from Python 3.13:

ASCII           chars=13 ascii=13 latin-1=13 utf-8=13 utf-16-le=26 utf-32-le=52
Accented Latin  chars=19 ascii=n/a latin-1=19 utf-8=23 utf-16-le=38 utf-32-le=76
CJK             chars= 5 ascii=n/a latin-1=n/a utf-8=15 utf-16-le=10 utf-32-le=20
Emoji           chars= 3 ascii=n/a latin-1=n/a utf-8=12 utf-16-le=12 utf-32-le=12

Here is the same data as a table, with what each row teaches.

Sample	Code points	ASCII	Latin-1	UTF-8	UTF-16	UTF-32
Hello, world!	13	13	13	13	26	52
Café déjà vu, señor	19	n/a	19	23	38	76
你好，世界	5	n/a	n/a	15	10	20
😀👍🏽	3	n/a	n/a	12	12	12

Read the table row by row.

ASCII-only text. UTF-8 is exactly the size of ASCII, and UTF-16 doubles it. This is the single biggest reason UTF-8 dominates the web: HTML, CSS, JavaScript, JSON, HTTP headers and URLs are mostly ASCII punctuation and keywords, even in pages written in other languages.

Accented Latin. Each accented letter costs two bytes in UTF-8 against one in Latin-1. The 19-character sample has four accented letters (é, é, à, ñ), so UTF-8 comes to 19 + 4 = 23 bytes. Latin-1 cannot represent anything beyond its 256 values, but for this text it is the most compact.

CJK. Chinese, Japanese and Korean ideographs and Hangul syllables sit in the BMP above U+0800, so UTF-8 spends three bytes on each and UTF-16 only two. This is the strongest argument for UTF-16 on storage cost, a third smaller for pure CJK text. Notice that the full-width comma ， also takes three bytes, so all five characters cost three bytes each. Real documents mix in ASCII markup and spaces, which shifts the balance back toward UTF-8, as we discuss below.

Emoji. Everything outside the BMP costs four bytes in all three Unicode encodings. The sample 😀👍🏽 has three code points, but the second glyph is a thumbs-up (U+1F44D) followed by a skin tone modifier (U+1F3FD), a separate code point. A user perceives two symbols, a library counts three code points, and JavaScript reports a length of six, because each of the three code points is a surrogate pair.

One string, many lengths

That last point deserves its own demonstration. Asking "how long is this string?" has at least four valid answers, depending on the unit.

Text	Bytes (UTF-8)	UTF-16 code units	Code points	What a user sees
Hello, world!	13	13	13	13
Café déjà vu, señor	23	19	19	19
你好，世界	15	5	5	5
😀👍🏽	12	6	3	2
👨‍👩‍👧	18	8	5	1

The last row is a family emoji built from three person emoji joined by invisible zero-width joiners (U+200D). That is five code points rendered as one image. The user-perceived unit is called a grapheme cluster, and the number of them in the string requires an algorithm from Unicode Standard Annex 29, which modern runtimes expose through APIs such as Intl.Segmenter in JavaScript. All the counts in the table were measured by code on the sample strings, in Python and Node.js.

String length and indexing across languages

Each language picked its own internal string model, and length means something different depending on that choice. Here is what a quick test reveals.

const text = "Naïve café 你好 😀";
const encoder = new TextEncoder();
console.log("UTF-16 code units:", text.length);
console.log("Code points:", [...text].length);
console.log("UTF-8 bytes:", encoder.encode(text).length);

const grin = "😀";
console.log(grin.charCodeAt(0).toString(16), grin.charCodeAt(1).toString(16));
console.log(grin.codePointAt(0).toString(16));
console.log(JSON.stringify(grin.slice(0, 1)), grin.slice(0, 1).isWellFormed());

Output from Node.js 22:

UTF-16 code units: 16
Code points: 15
UTF-8 bytes: 24
d83d de00
1f600
"\ud83d" false

The string has 15 characters in the sense of code points, but length reports 16, because the emoji occupies two UTF-16 code units. Slicing the string at index 1 produces a lone high surrogate, which is not valid Unicode text, and isWellFormed() reports false. The spread operator [...text] iterates by code point, which is why it returns 15.

Language	What len/length counts	Internal form
JavaScript	UTF-16 code units	UTF-16 (engines may store Latin-1 text compactly, invisibly)
Java, C#, Kotlin	UTF-16 code units (char)	UTF-16
Python 3	Code points	Flexible; the interpreter picks a width per string
Rust	len() counts UTF-8 bytes; chars().count() counts code points	UTF-8
Go	len() counts bytes; utf8.RuneCountInString counts code points	Bytes, conventionally UTF-8
Swift	count counts grapheme clusters	Unicode-aware
C and C++ (char)	Bytes	Whatever the bytes are

Two practical rules fall out of this table.

First, never use a length as a proxy for storage size. A database column limited to 255 bytes and a form field limited to 255 characters can disagree for any text that is not ASCII. Decide which unit your limit uses and measure that unit.

Second, never slice text in the middle of something that should stay together. In UTF-16 languages, slicing can split a surrogate pair. In byte-oriented languages, slicing can split a multi-byte UTF-8 sequence. At the grapheme level, slicing can separate a base letter from its combining accent or an emoji from its skin tone. When you truncate user text for display, use a segmentation API or add an ellipsis only at a safe boundary.

Precomposed versus decomposed text

One more source of length surprises is normalization. The letter é can be a single code point, U+00E9, or two: a plain e followed by a combining acute accent, U+0301. They render identically but are different strings.

import unicodedata
a = "é"
b = "é"
print(len(a), len(b), a == b)
print(unicodedata.normalize("NFC", b) == a)
print(a.encode("utf-8").hex(" "), b.encode("utf-8").hex(" "))

Output:

1 2 False
True
c3 a9 65 cc 81

If you compare strings, look them up in a database or hash them for a cache key, normalise both sides to the same form first. The common choices are NFC (compose where possible) and NFD (decompose). NFC is the usual default for text stored and exchanged on the web. Macs have historically stored file names in a decomposed form, which causes surprising mismatches when a name created on one system is compared on another.

Why platforms use UTF-16 inside and UTF-8 outside

You will notice a pattern: JavaScript, Java, C# and the Windows API use UTF-16 internally, while files, web pages and network protocols overwhelmingly use UTF-8. This looks inconsistent but has a historical explanation.

Internal UTF-16 is an inheritance. When Java, JavaScript and Windows NT were designed in the 1990s, Unicode was a 16-bit standard and a 16-bit char seemed to be a clean fixed-width solution. Those platforms committed to 16-bit strings in their public APIs, in compiled code and in billions of lines of application logic. Changing the unit would break array indexing, charAt, interop and binary compatibility. So they kept 16-bit units and extended them with surrogate pairs when Unicode grew.

External UTF-8 is a convenience. UTF-8 became the dominant encoding for interchange because it is ASCII compatible, has no byte order, is robust against damage and is compact for the Latin-heavy markup that makes up the web. When the two meet, a boundary conversion happens: a browser decodes UTF-8 bytes from the network into UTF-16 strings for JavaScript, and encodes outgoing strings back to UTF-8. The web platform standardised on UTF-8 for new work, and the WHATWG Encoding Standard states that new protocols and formats should use UTF-8.

The practical consequences are these:

• Convert at the edge, not in the middle. Decode bytes to strings when data enters your program, work with strings internally, and encode when data leaves.
• Expect a conversion cost, but not a big one. For typical request sizes, transcoding is cheap compared with network and parsing work. Measure before optimising.
• Do not leak the internal form. If a Java or JavaScript program writes its internal UTF-16 to a file without a deliberate choice, you get UTF-16 files with all the byte order and compatibility problems that brings.

Some platforms are moving in the other direction. Newer languages such as Rust and Swift use UTF-8 internally, and some runtimes store pure-Latin text compactly while presenting a UTF-16 interface. Python picks a per-string width internally and hides it from you. Check your runtime's documentation if memory or speed on large text matters, because these are implementation details that can change between versions.

Storage trade-offs and random access

Which encoding is smallest depends on your text. Here is a simple model for choosing, based on how many bytes each class of character costs.

Character class	UTF-8	UTF-16	UTF-32
ASCII (U+0000 to U+007F)	1	2	4
Latin, Greek, Cyrillic, Hebrew, Arabic (U+0080 to U+07FF)	2	2	4
Most CJK, Indic scripts, Thai, symbols (U+0800 to U+FFFF)	3	2	4
Emoji and rare scripts (U+10000 and above)	4	4	4

By this table, UTF-8 wins for English and most European text, ties for Greek, Cyrillic, Hebrew and Arabic when there are no spaces or digits, and loses by a third for scripts that live above U+0800. UTF-16 can win on pure East Asian text. UTF-32 never wins on size.

The "pure" qualification matters. Real text includes spaces, punctuation, digits, line breaks and, in web formats, ASCII markup. Each of those costs one byte in UTF-8 and two in UTF-16. A Chinese web page is full of ASCII tags and attribute names, and an XML file in Japanese has English element names. In practice, UTF-8 is often smaller than UTF-16 even for text in languages where UTF-16 is theoretically cheaper per character. Once you apply gzip or Brotli, the gap narrows further, because compressors exploit the redundancy in the encoded bytes.

To test your own content, encode a representative sample both ways and compare. The following Node.js snippet does it for a string you provide:

const sample = "你好，世界 <p class=\"intro\">欢迎</p>";
const utf8 = new TextEncoder().encode(sample).length;
const utf16 = sample.length * 2;
console.log({ utf8, utf16 });

Because TextEncoder always produces UTF-8, and UTF-16 uses exactly two bytes per code unit, the comparison needs no extra library. On this sample, which mixes CJK text with HTML tags, the script printed { utf8: 43, utf16: 58 }, so the ASCII markup made UTF-8 clearly smaller even though the CJK characters alone would favour UTF-16.

Random access versus variable width

The other classic trade-off is access pattern. With a fixed-width encoding, finding the nth character is arithmetic: offset = n times width. With variable-width encodings, you must scan from the start, or from a known position.

• UTF-32 gives random access by code point in constant time.
• UTF-16 gives constant-time access by code unit, but not by code point, because surrogate pairs vary the width. For BMP-only text it behaves as fixed width, which is why bugs with surrogate pairs hide for years and then appear when the first emoji arrives.
• UTF-8 gives constant-time access by byte, but not by character.

In practice, this matters less than people expect. Real programs rarely need "the 10,000th character". They scan from start to end, search for substrings, split on delimiters and compare. UTF-8's self-synchronising property makes byte-level searching for a known substring correct: because a valid UTF-8 sequence can never appear inside another valid sequence's continuation bytes, finding the bytes of needle in the bytes of haystack finds real character matches. Text editors and databases that need positional access build indexes, such as ropes or line tables, rather than switching to a wasteful fixed width.

Real-world use cases

Web pages and HTTP. Serve HTML as UTF-8 and say so, with the Content-Type: text/html; charset=utf-8 header, and with <meta charset="utf-8"> as a fallback near the top of <head>. CSS and JavaScript files are also best served as UTF-8. See our guides to HTML formatting and semantic markup and CSS formatting and architecture for the surrounding practices.

JSON APIs. The JSON specification, RFC 8259, requires UTF-8 for JSON exchanged between systems. If you control a JSON API, the answer is already decided. The related question is whether to escape non-ASCII characters as \uXXXX sequences, which makes the payload ASCII-safe but larger. Our guide to JSON minification and payload size measures that trade-off, and the JSON formatting guide covers syntax.

Source code. Most compilers and interpreters expect UTF-8 source today. Identifiers and string literals can contain non-ASCII characters, but mixing encodings inside a repository produces unreadable diffs. Our guide to JavaScript code formatting best practices touches on consistent files, and a diff tool, as described in our guide to comparing files, will show invisible differences such as BOMs once you know to look.

Databases. Choose UTF-8 for text columns, and watch for vendor quirks. A well-known one is in MySQL, where the character set named utf8 historically stores at most three bytes per character, which cannot represent emoji, while utf8mb4 is full UTF-8. Always check which setting your database actually applies to the connection, the table and the column, because a mismatch at any level can truncate or corrupt text. Our SQL formatting guide covers the query side.

Email. Message bodies and headers are transported using MIME, which labels each part with a charset parameter and a transfer encoding, such as quoted-printable or Base64, to make 8-bit data safe for older mail servers. Non-ASCII header values, such as a subject line, use encoded words. UTF-8 is the right charset for new mail, but you must still declare it. A missing or wrong declaration is how an accented name in a subject turns into question marks.

Files and CSV exports. Text files carry no metadata. When you export CSV for a spreadsheet program, the program has to guess the encoding. Some spreadsheet tools assume a legacy code page unless the file starts with a UTF-8 BOM, which is the one place where adding a BOM can be the pragmatic choice. We discuss it below.

Filenames and paths. Operating systems differ: some store filenames as byte sequences without any declared encoding, and others as UTF-16 code units. Treat filenames with care when you move archives between systems.

XML and SVG. XML declares its encoding in the prolog, and the default when no declaration is present is UTF-8 (or UTF-16 when a BOM says so). See our guides to XML formatting and structure and SVG formatting and editing for details on well-formed documents.

URLs. Non-ASCII characters in URLs are percent-encoded from their UTF-8 bytes. The euro sign becomes %E2%82%AC, and a space becomes %20. The DevUtilX UTF-8 Encoder, described next, shows this form directly.

Common mistakes and pitfalls

Mistake 1: Declared encoding versus actual bytes

A page that says charset=utf-8 but is saved as Windows-1252, or the reverse, will show garbled text. The declaration is a claim, and the bytes are the truth. Mismatches usually arise when a developer edits a file in an editor with a different default, or when a database dumps data in the connection's encoding rather than the table's.

Mistake 2: Relying on sniffing

When no encoding is declared, browsers and tools guess. The HTML parsing rules describe a sequence of signals in order of priority: a BOM at the start of the file, then the Content-Type charset from the HTTP header, then a <meta> declaration found by scanning the first part of the document (the specification's prescan looks at the first 1024 bytes), and finally a locale-specific default. Because the meta tag must appear early, a charset declaration buried after a long <title> can be missed. Put <meta charset="utf-8"> as the first element inside <head>, and set the HTTP header too, since the header wins over the meta tag.

Mistake 3: Assuming Latin-1 and Windows-1252 are the same

They differ in the range 0x80 to 0x9F. A document that contains curly quotes or the euro sign decodes correctly as Windows-1252 and as garbage control characters in strict Latin-1. Browsers resolve the ambiguity by treating the iso-8859-1 label as Windows-1252, but Python's latin-1 codec is strict, as the mojibake section shows.

Mistake 4: Treating UTF-16 as ASCII-compatible

Reading UTF-16 text as ASCII or UTF-8 produces alternating characters and null bytes, because every ASCII character is followed or preceded by a zero byte. A tool that stops at the first zero byte sees one letter. If a file seems to contain only the first character of every word, check for UTF-16 and look for the ff fe or fe ff bytes at the start.

Mistake 5: Forgetting about the BOM

A UTF-8 BOM breaks tools that do not expect it. A shell script starting with ef bb bf can fail with a cryptic "command not found" on its first line. A JSON parser may reject the document, because a BOM is not valid JSON whitespace. A CSV header may gain an invisible first character, so a lookup for id fails because the key is actually id:

data = b"\xef\xbb\xbfid,name\n1,Ada\n"
print(repr(data.decode("utf-8").splitlines()[0]))
print(repr(data.decode("utf-8-sig").splitlines()[0]))

Output:

'id,name'
'id,name'

The utf-8-sig codec in Python strips a BOM on read and adds one on write. In Node.js you strip it manually, for example with text.replace(/^/, "").

Mistake 6: Splitting surrogate pairs and multi-byte sequences

Truncating a string at a fixed number of UTF-16 units or UTF-8 bytes can cut an emoji or a character in half. The result is a lone surrogate or an invalid byte sequence. Python refuses to encode a lone surrogate to UTF-8:

UnicodeEncodeError: 'utf-8' codec can't encode character '\ud83d' in position 0: surrogates not allowed

JavaScript is more forgiving, which is worse. encodeURIComponent throws URIError: URI malformed for a lone surrogate, while TextEncoder silently replaces it with U+FFFD. In both cases, the data is damaged by the time someone notices.

Mistake 7: Using the wrong unit for limits

Validating a username as "at most 20 characters" with a byte-length check rejects a valid 8-character Greek name. Validating a column limit with a code point count lets an emoji-heavy value overflow a byte-limited column. Pick the unit deliberately, and document it.

Mistake 8: Silent replacement and lossy conversion

Many conversions have an error policy. Python's errors="replace" writes ? for characters that cannot be encoded and U+FFFD for bytes that cannot be decoded. That keeps programs running and silently destroys data. In a data pipeline, prefer strict errors during development so that you find bad input, and handle failures explicitly in production.

Mistake 9: Double encoding

Text that was already UTF-8 is decoded as Latin-1 or Windows-1252 and then encoded as UTF-8 again. Each pass multiplies the damage, and the result grows longer. It happens when two layers each believe they must convert, for example a database driver and an application. The mojibake section shows how it looks and how to undo a single round.

Mistake 10: Mixing normalization forms

Two visually identical strings fail an equality check because one is composed and the other decomposed. This hits search, deduplication, passwords, usernames and file names. Normalise on input, and keep one form in storage.

Mojibake: what mismatched pairs look like

Mojibake is the garbled text that results when bytes are decoded with the wrong encoding. The Mojibake article on Wikipedia catalogues many examples. Learning to recognise the patterns lets you identify both the writer's and the reader's encoding at a glance.

This script produces the classic mismatches from one source string:

text = "Café – 你好"
raw = text.encode("utf-8")
print("UTF-8 bytes read as Latin-1:", raw.decode("latin-1"))
print("Latin-1 bytes read as UTF-8:", "Café".encode("latin-1").decode("utf-8", errors="replace"))
print("Repaired:", "Café".encode("latin-1").decode("utf-8"))
print("Double-encoded:", "é".encode("utf-8").decode("cp1252").encode("utf-8").decode("cp1252"))
print("UTF-16 LE read as UTF-8:", repr("Hi".encode("utf-16-le").decode("utf-8")))
print("UTF-16 LE read as BE:", "Hi".encode("utf-16-le").decode("utf-16-be"))

Output:

UTF-8 bytes read as Latin-1: Café â ä½ å¥½
Latin-1 bytes read as UTF-8: Caf�
Repaired: Café
Double-encoded: é
UTF-16 LE read as UTF-8: 'H\x00i\x00'
UTF-16 LE read as BE: 䠀椀

Note that the first line also contains invisible C1 control characters that the terminal does not show, which is why the en dash appears as a lone â followed by a gap.

The patterns, and what they mean

What you see	Probable cause	Direction of the fix
é in place of é, ñ in place of ñ	UTF-8 bytes decoded as Latin-1 or Windows-1252	Decode as UTF-8 instead
’ or – in place of a curly quote or dash	UTF-8 bytes decoded as Windows-1252	Decode as UTF-8 instead
Caf� with the diamond or a box	Legacy bytes (such as Latin-1 e9) decoded as UTF-8 and failed	Decode as Latin-1 or Windows-1252 instead
é and similarly long strings	Double encoding	Reverse each wrong decode once, then fix the pipeline
Alternate characters and \x00 bytes	UTF-16 read as an 8-bit encoding	Decode as UTF-16
Unexpected CJK characters from plain English	UTF-16 byte order mismatch, or 8-bit text decoded as UTF-16	Check the BOM and the endianness
Question marks in place of letters	Lossy conversion on write: the characters were replaced and are gone	Restore from the original source
Characters shown as boxes	The font lacks glyphs, though the bytes may be fine	Check the font, not the encoding

The first four rows are reversible, because they only rearrange bytes, so the original information still exists. The Repaired: line above shows the repair: encode the garbled string back to the encoding that was wrongly used to decode it, and decode the resulting bytes with the correct one. Question marks, by contrast, are usually permanent loss, since a lossy conversion has thrown away the original character. This distinction tells you whether a rescue is possible.

A mojibake repair helper

For a quick diagnosis, this Python function tries a few likely repairs and returns the first that decodes cleanly:

def try_repair(s):
    for wrong in ("latin-1", "cp1252"):
        try:
            return s.encode(wrong).decode("utf-8")
        except (UnicodeEncodeError, UnicodeDecodeError):
            continue
    return None

print(try_repair("Café"))

Output:

Café

Use this only for diagnosis. For production data, find the real source of the mismatch and fix it there, because a heuristic repair can miss cases and make text worse.

Step by step: inspect text with the DevUtilX tools

When you need to see what a string looks like under the hood, two DevUtilX tools help. Both run in your browser, so the text you paste is processed locally and nothing is uploaded.

See code points and escapes with Text to Unicode

The Text to Unicode converter turns characters into \uXXXX escape sequences and back. These are the notation used in JavaScript, Java and JSON source.

1. Open the Text to Unicode tool.
2. Paste or type your text, for example Café 你 😀.
3. Run the conversion and read the output. Each character appears as a four-digit hex escape.
4. Look at the emoji. It appears as two escapes, \ud83d \ude00, because each escape stands for one UTF-16 code unit, so characters outside the BMP show up as a surrogate pair.
5. To go the other way, paste a string of \u escapes and convert it back to text. This is handy for decoding escaped strings found in logs and API responses.

This is also an effective way to confirm what an invisible character is. A stray  at the start of your input is a BOM, and ​ is a zero-width space. If two strings look the same but compare unequal, convert both and compare the escapes. A combining accent will show up as an extra ́.

See bytes with the UTF-8 Encoder

The UTF-8 Encoder shows the UTF-8 form of your text as percent-encoded bytes, the form used in URLs and query strings.

1. Open the UTF-8 Encoder.
2. Paste your text, for example Café 你 😀.
3. Run the encoder.
4. Read the output: Caf%C3%A9%20%E4%BD%A0%20%F0%9F%98%80. The é becomes two bytes (%C3%A9), the Chinese character three (%E4%BD%A0) and the emoji four (%F0%9F%98%80), matching the table of UTF-8 ranges earlier in this guide.
5. Count the %XX groups per character to see the byte widths yourself.

Plain ASCII letters and digits stay as they are, and the space becomes %20, since the percent-encoded form is designed for URLs. To reverse a percent-encoded string back into text, use a UTF-8 decoder rather than this page.

Using both tools on the same text gives you a quick view of the two layers: the first shows the code units, and the second shows the bytes. If the escapes look right but the bytes do not, the problem is in your encoding step. If the escapes are already wrong, the problem happened before.

Code examples you can run

All outputs below were produced by running the code on the sample text in this article, with Node.js 22 and Python 3.13.

JavaScript: encode, decode and inspect

function utf8Bytes(cp) {
  if (cp < 0x80) return [cp];
  if (cp < 0x800) return [0xc0 | (cp >> 6), 0x80 | (cp & 0x3f)];
  if (cp < 0x10000) return [0xe0 | (cp >> 12), 0x80 | ((cp >> 6) & 0x3f), 0x80 | (cp & 0x3f)];
  return [0xf0 | (cp >> 18), 0x80 | ((cp >> 12) & 0x3f), 0x80 | ((cp >> 6) & 0x3f), 0x80 | (cp & 0x3f)];
}
for (const ch of ["A", "é", "你", "😀"]) {
  const cp = ch.codePointAt(0);
  console.log(ch, "U+" + cp.toString(16).toUpperCase().padStart(4, "0"),
    utf8Bytes(cp).map((b) => b.toString(16)).join(" "));
}

Output:

A U+0041 41
é U+00E9 c3 a9
你 U+4F60 e4 bd a0
😀 U+1F600 f0 9f 98 80

This hand-written encoder follows the table of byte patterns exactly. In real code, use TextEncoder and TextDecoder, which are built in to browsers and Node.js. The MDN page on TextEncoder documents the API. TextEncoder always produces UTF-8, while TextDecoder can decode many labels, including windows-1252 and utf-16le:

const encoder = new TextEncoder();
const bytes = encoder.encode("Café €");
console.log(new TextDecoder("utf-8").decode(bytes));
console.log(new TextDecoder("windows-1252").decode(bytes));
console.log(new TextDecoder("utf-16le").decode(Uint8Array.from([0x48, 0, 0x69, 0])));

Output:

Café €
Café €
Hi

The second line is mojibake produced on purpose: UTF-8 bytes decoded as Windows-1252. In Node.js, Buffer offers similar conversions. Buffer.from("é", "latin1") gives one byte, e9, Buffer.from("é", "utf8") gives c3 a9, and Buffer.from("é", "utf16le") gives e9 00.

Python: bytes are not text

Python 3 separates text (str, a sequence of code points) from bytes (bytes). The boundary is .encode() and .decode(), and the safest habit is to always name the encoding:

# Reading and writing files: always pass encoding
with open("notes.txt", "w", encoding="utf-8", newline="\n") as f:
    f.write("Café 你好\n")

with open("notes.txt", "r", encoding="utf-8") as f:
    print(f.read().strip())

# Decode bytes strictly, so bad input fails loudly
data = "Café".encode("utf-8")
print(data.decode("utf-8"))

Without an explicit encoding, Python's open uses a locale-dependent default, which can differ between machines, and especially between Windows and Linux. That single omission explains a large share of "works on my machine" encoding bugs. Pass encoding="utf-8" every time, and use "utf-8-sig" when reading files that may start with a BOM.

A third view: the command line

If you work in a terminal, iconv converts between encodings and file makes a best guess at what a file contains:

iconv -f WINDOWS-1252 -t UTF-8 legacy.txt > utf8.txt
file -i utf8.txt

The -f flag names the source encoding and -t the target. Naming the wrong source silently corrupts the result, so confirm the true source encoding by inspecting a few bytes with a hex dump before you convert a whole directory.

Choosing an encoding: a decision table

For nearly every new project, the answer is the same. Use this table to confirm it for your situation.

Situation	Recommended encoding	Why
Web pages, HTML, CSS, JavaScript files	UTF-8	The web standard, required for new HTML documents
JSON, YAML, TOML, config files	UTF-8	JSON requires UTF-8 for interchange, and the rest assume it
REST and GraphQL APIs	UTF-8	Interoperable, ASCII-compatible, no byte order
Databases	UTF-8 (utf8mb4 in MySQL)	Full Unicode, compact for mixed content
Email	UTF-8 with a declared charset	Works with MIME, covers all languages
Source code	UTF-8 without BOM	Tool compatibility, clean diffs
Log files and CSV for Unix tools	UTF-8 without BOM	Scripting-friendly
CSV for older spreadsheet software	UTF-8 with BOM (test it)	Some programs need the signature to pick UTF-8
In-memory strings in JavaScript, Java, C#	Whatever the platform uses (UTF-16)	Do not fight the runtime, convert at boundaries
Windows API calls (wide-character functions)	UTF-16LE	The native form of those APIs
Pure CJK storage where bytes are scarce	Consider UTF-16, after measuring	Smaller than UTF-8 for text with few ASCII characters
Algorithms needing code point indexing	UTF-32 as a temporary form	Simple access, but memory-hungry
Existing systems with legacy data	Keep stable until migrated, then convert	Mixed encodings are the worst outcome
New protocols and file formats	UTF-8	The WHATWG Encoding Standard says new formats should use it

The pattern is consistent: UTF-8 for anything that is stored, sent or shared, and the platform's native form for in-memory strings. The only strong exceptions are cases where you must match an external API that mandates another encoding.

Migrating from legacy encodings to UTF-8

Migrations fail when people convert without first knowing the source encoding. Follow this order of work.

1. Inventory. List every place text is stored or exchanged: database tables and connections, files, message queues, third-party feeds, and the HTTP layer. Record the declared encoding for each, then verify it by sampling real bytes.
2. Identify the true source encoding. Do not trust labels. Test a sample of values: if decoding as UTF-8 succeeds in strict mode on all the non-ASCII data, it is probably already UTF-8, since random legacy text rarely forms valid multi-byte sequences by accident. If it fails, try the likely code page for your users, such as Windows-1252 for Western Europe.
3. Back up before touching anything. Conversions are hard to undo, and a wrong conversion can destroy information.
4. Convert in a test copy. Run the conversion with strict error handling, so unencodable or invalid bytes stop the process instead of being replaced by ?.
5. Handle double-encoded data. If sampling shows strings such as é, the data was already UTF-8 bytes stored in a column that was labelled Latin-1. A naive conversion would double-encode them. Fix such columns by relabelling rather than converting, or by reversing one round of damage as shown earlier.
6. Update every layer. Database column and table character sets, the connection setting used by your driver, the application's file reading and writing, HTTP headers, HTML meta tags and email templates. A single layer left behind will reintroduce mojibake.
7. Check limits. A column sized in bytes may need to grow, since text that fitted in Latin-1 occupies more space in UTF-8. Indexes can also hit size limits when you widen columns.
8. Normalise. Pick NFC for stored text and apply it as part of the migration.
9. Verify. Compare row counts, spot-check names with accents and non-Latin scripts, and test round trips through the entire stack: form, database, API, export.
10. Monitor after cutover. Search logs and a sample of new records for U+FFFD, Ã sequences and ? in unexpected places.

A strict conversion in Python makes the policy explicit:

raw = "Café".encode("cp1252")
text = raw.decode("cp1252")        # strict by default: raises on invalid bytes
out = text.encode("utf-8")
print(out.hex(" "))

Output:

43 61 66 c3 a9

The default strict error handling is your friend during a migration. Switch to a lenient policy only when you have decided what to do with the failures, and log every replacement.

Performance, security and accessibility

Performance

For most programs, encoding is not a performance bottleneck. The areas where it matters are these:

• Network and storage size. UTF-8 usually wins for web content because of ASCII markup. Compression narrows differences further.
• Transcoding at boundaries. Converting between UTF-8 and UTF-16 costs CPU proportional to text size. Avoid converting the same large buffer repeatedly, and keep data in bytes if you do not need to inspect it.
• Scanning and indexing. Fixed-width encodings give constant-time indexing, but most real workloads scan sequentially, where variable width costs little.
• Memory. A UTF-32 buffer holds four bytes for every code point, which is expensive for large text. Do not use it as a storage format.

Measure with your own data and runtime before making a decision, since these are implementation details that vary by language and version.

Security

Encoding mismatches are a long-standing source of vulnerabilities, not only cosmetic bugs.

• Overlong sequences and validation order. If a filter checks for ../ in one representation and the system later decodes a different representation, an attacker can slip the sequence past the filter. Decode and normalise first, validate second, and use a strict UTF-8 decoder that rejects overlong forms.
• Lookalike characters. Many scripts contain letters that look like Latin ones, such as Cyrillic а and Latin a. Attackers use them in domains and usernames. Normalisation does not remove this, so apply additional checks on identifiers where impersonation matters.
• Invisible characters. Zero-width spaces, joiners and the right-to-left override can hide content or reorder how code appears. Review tools that show escapes make these visible, which is a good reason to inspect suspicious strings with the Text to Unicode tool.
• Length limits and truncation. Truncating a validated string after validation can produce a different, invalid or dangerous string. Truncate before validating or re-validate after.
• Unicode and case-insensitive comparisons. Compare identifiers after normalisation and with proper case folding, not with simple lower-casing, to avoid duplicate accounts that differ only by encoding.
• Escaping on output. Encoding does not replace output escaping. Data embedded in HTML, SQL or shell commands still needs context-specific escaping.

Accessibility and internationalisation

Correct encoding is a prerequisite for accessibility. Screen readers announce garbled text literally, so mojibake makes content unusable to people who rely on assistive technology. Declare the document language with the lang attribute along with the encoding, because pronunciation and font selection depend on it. Do not cut user-visible text in the middle of a grapheme cluster, since that can leave a broken accent or emoji sequence. Make sure fonts cover the scripts you display, and offer a fallback, because missing glyphs show as empty boxes even when the bytes are correct. For text that mixes left-to-right and right-to-left scripts, use the dir attribute and the Unicode bidirectional algorithm in the browser rather than inserting control characters by hand.

Troubleshooting: symptom, cause, fix

Symptom: é or ’ appears where accents or quotes should be

Cause. UTF-8 bytes were decoded as Latin-1 or Windows-1252.
Fix. Make the reader use UTF-8: set the HTTP Content-Type charset, the HTML meta tag, the database connection charset, or the encoding argument of the file read. For stored data that is already damaged, reverse the wrong decode once, as in the repair example.

Symptom: black diamonds with question marks, or �, in place of letters

Cause. The source is a legacy encoding such as Latin-1 or Windows-1252, and the reader tried UTF-8. Invalid byte sequences become the replacement character U+FFFD.
Fix. Decode with the right legacy encoding, then re-save as UTF-8. If the data has already been saved after the replacement, the original characters are lost and must come from the source.

Symptom: plain question marks in a database

Cause. A conversion to a narrower character set replaced characters it could not represent. The column or connection is probably Latin-1 or a three-byte utf8.
Fix. Change the column, table and connection to full UTF-8, then reload the data from its original source. The characters cannot be recovered from the question marks.

Symptom: emoji cannot be saved, or the save truncates the text at the emoji

Cause. The database's utf8 setting stores at most three bytes per character, and emoji need four.
Fix. Switch to the full four-byte UTF-8 character set, and check the connection setting as well as the column.

Symptom: the first CSV column header has an invisible prefix

Cause. A UTF-8 BOM at the start of the file was read as part of the first field name.
Fix. Read with a BOM-aware decoder such as Python's utf-8-sig, or strip  explicitly. When you export for a Unix pipeline, write the file without a BOM.

Symptom: every character is followed by a null byte, or text appears as spaced letters

Cause. UTF-16 data is being read as an 8-bit encoding.
Fix. Decode as UTF-16, using the BOM or a known byte order. Convert the file to UTF-8 if it feeds Unix tools.

Symptom: unexpected Chinese or Japanese characters appear from English input

Cause. 8-bit text was decoded as UTF-16, or UTF-16 was decoded with the wrong byte order, which pairs bytes into 16-bit values that land in CJK ranges. The earlier example turned Hi into 䠀椀.
Fix. Check the BOM, then pick the correct label (utf-16le or utf-16be).

Symptom: length or a counter is off for emoji or accented text

Cause. The language counts UTF-16 code units or bytes, and the text contains surrogate pairs, multi-byte characters or combining marks.
Fix. Decide the unit you want. Use code point iteration, Intl.Segmenter for grapheme clusters, or normalise first. Document the limit.

Symptom: strings that look identical are not equal

Cause. One is precomposed (é as U+00E9), the other decomposed (e plus U+0301), or one contains an invisible character.
Fix. Normalise both to NFC before comparing, and inspect the escapes in the Text to Unicode tool to find hidden characters.

Symptom: UnicodeDecodeError or UnicodeEncodeError

Cause. The data does not match the declared encoding, or the target encoding cannot represent a character.
Fix. Read the error message, which names the byte or character and its position. Look at the bytes around that position to identify the real encoding. Fix the source or choose a wider target such as UTF-8, rather than adding errors="ignore", which hides the problem.

Symptom: a lone surrogate breaks JSON or a network call

Cause. A string was sliced through a surrogate pair, or came from an API that allows invalid UTF-16.
Fix. Use isWellFormed() and toWellFormed() where available, or slice by code point. Clean the data at the boundary instead of passing it on.

Best practices checklist

• Use UTF-8 everywhere you store, send or share text. Treat any other choice as something that needs a written reason.
• Declare the encoding explicitly: HTTP headers, <meta charset="utf-8"> as the first element in <head>, XML prolog, database and connection settings, MIME charset, and the encoding argument in file APIs.
• Decode at the boundary and encode at the boundary. Inside the program, work with strings, not bytes.
• Never guess silently. Fail loudly on invalid input in development, and log replacements in production.
• Write UTF-8 without a BOM, except where a specific consumer needs it, and read files with a BOM-aware decoder.
• Normalise text to one form, usually NFC, before comparing, indexing or storing.
• Measure in the right unit. Bytes for storage and network limits, code points for text algorithms, grapheme clusters for what the user sees.
• Do not truncate through the middle of a character or cluster. Use segmentation APIs when you shorten text for display.
• Use full four-byte UTF-8 in databases and check collations, indexes and column sizes.
• Test with hostile samples: accented Latin, CJK, right-to-left text, emoji with modifiers, combining marks and an unpaired surrogate. Put them in your fixtures.
• Keep a strict UTF-8 validator at trust boundaries so overlong and malformed sequences are rejected.
• Convert legacy data once, deliberately, and keep the original until you have verified the result.
• Use proper tools to inspect: a hex view, the Text to Unicode converter for code units, and the UTF-8 Encoder for bytes.

Comparison of alternatives

Encoding	Bytes per character	ASCII compatible	Byte order issue	Best for	Main drawback
ASCII	1	Yes (it is ASCII)	None	Protocol tokens, English-only identifiers	Cannot represent accents or other scripts
ISO-8859-1	1	Yes	None	Reading old Western European data	Only 256 characters, no euro sign
Windows-1252	1	Yes	None	Reading old Windows documents	Western European only, misleadingly labelled
Shift_JIS, GBK, Big5, EUC-KR	1 or 2	Mostly	None	Maintaining legacy East Asian systems	Language-specific, tricky to detect
UTF-8	1 to 4	Yes	None	Files, web, APIs, databases, email	Three bytes for most CJK, no constant-time indexing
UTF-16	2 or 4	No	Yes	In-memory strings on JavaScript, Java, .NET and Windows	Surrogate pairs, byte order, no ASCII compatibility
UTF-32	4	No	Yes	Short-lived buffers needing code point indexing	Four times the size of ASCII text

Note that "ASCII compatible" here means that pure ASCII text has identical bytes in that encoding. That is why you can open an ASCII file as UTF-8 safely, but not as UTF-16.

FAQ

Is ASCII the same as UTF-8?
No, but ASCII is a subset of UTF-8. Every ASCII byte sequence is valid UTF-8 with the same meaning. UTF-8 then adds multi-byte sequences for all other characters, so UTF-8 text with accents is not valid ASCII.

Is Unicode an encoding?
No. Unicode is a character set that assigns numbers (code points) to characters. UTF-8, UTF-16 and UTF-32 are the encodings that turn those numbers into bytes. People often say "Unicode" when they mean UTF-16, because that is what some Windows tools call it, which adds to the confusion.

Why does "😀".length return 2 in JavaScript?
JavaScript strings are sequences of UTF-16 code units, and length counts those units. The emoji is outside the BMP, so it takes a surrogate pair of two units. Use [...text].length or Array.from(text).length to count code points, and Intl.Segmenter for user-perceived characters.

Should I use UTF-8 or UTF-16 for my database?
Use UTF-8 unless your database platform or a strong storage reason says otherwise. For mixed content with ASCII, such as names, URLs, markup and numbers, UTF-8 is usually the same size or smaller. In MySQL, use utf8mb4.

Does UTF-8 use more space than UTF-16?
It depends on the text. For ASCII and most European text it uses less. For text made mostly of characters above U+0800, such as Chinese, Japanese, Korean or Hindi, it uses about a third more bytes per character. Mixed content with markup narrows or reverses that difference, and compression narrows it further. Test a sample of your own data.

Should I add a BOM to UTF-8 files?
Usually not. It is not needed, and it breaks many Unix tools, scripts and parsers. Add one only when a specific consumer, such as an older spreadsheet program, requires it to detect UTF-8, and test that consumer. Always read files with a decoder that tolerates a BOM.

What is the difference between Latin-1 and Windows-1252?
They agree on most of the 8-bit range, but Windows-1252 uses the 0x80 to 0x9F range for printable characters such as curly quotes, the en and em dashes and the euro sign, where Latin-1 has control characters. Web browsers treat the iso-8859-1 label as Windows-1252, so documents with that label still display correctly.

What is a surrogate pair?
It is how UTF-16 represents code points above U+FFFF using two 16-bit units: a high surrogate in the range D800 to DBFF followed by a low surrogate in the range DC00 to DFFF. Together they carry 20 bits of the code point. Surrogates are not characters by themselves, and a string containing an unpaired one is not valid Unicode text.

How do I fix text that shows é?
It is UTF-8 that was decoded as Latin-1 or Windows-1252. For the display, make the reader decode as UTF-8. For stored data, reverse one wrong decode by encoding the text back with the wrong encoding and decoding the bytes as UTF-8, then repair the pipeline that caused it.

Why do some platforms still use UTF-16 internally?
Java, JavaScript and Windows adopted 16-bit characters when Unicode was believed to fit in 16 bits. Changing the string unit later would break existing code and binary interfaces, so they extended it with surrogate pairs. At the edges, such as files and networks, they convert to UTF-8.

Can I detect an encoding reliably from the bytes?
Not with certainty. A BOM is a strong signal when present, and valid multi-byte UTF-8 in non-ASCII data is a strong sign of UTF-8, since random legacy text rarely forms valid sequences. Beyond that, detection libraries guess statistically, and they can be wrong on short texts. The reliable fix is to declare the encoding at the source.

How many bytes can a UTF-8 character use?
One to four, in the current definition, covering code points up to U+10FFFF. Older drafts allowed up to six bytes, but RFC 3629 restricts it to four.

Does encoding affect URLs?
Yes. Non-ASCII characters in the path or query string are sent as percent-encoded UTF-8 bytes. For example, é becomes %C3%A9. If a site decodes them with the wrong encoding, you see mojibake in parameters.

Is it safe to paste text into an online encoder?
Only if the tool runs locally. DevUtilX tools process your input in your browser, but avoid pasting passwords, tokens or personal data into any web page.

Conclusion

Character encoding looks like a maze, but the map is short. ASCII covers 128 English-centric characters in 7 bits. The legacy code pages stretch it to 8 bits in incompatible ways. Unicode assigns one number to every character, and UTF-8, UTF-16 and UTF-32 are three ways to turn those numbers into bytes. UTF-8 is ASCII-compatible, free of byte order problems and the right default for files, databases, APIs and email, while UTF-16 lives inside JavaScript, Java and Windows for historical reasons. Most bugs come from declaring the wrong encoding, guessing, or counting in the wrong unit, so declare explicitly, convert at the edges and measure in the unit you mean.

When you need to see the details, try the DevUtilX Text to Unicode converter to inspect code units and escapes, and the UTF-8 Encoder to see the bytes. Both run in your browser. To go further, read our guides on JSON minification and payload size, where escaping non-ASCII text changes the size of a response, and on comparing files and code diffs, which helps you spot invisible differences such as BOMs and mixed normalization.

Further reading

• Character encoding – Wikipedia
• UTF-16 – Wikipedia
• Character encodings: Essential concepts – W3C Internationalization
• RFC 3629: UTF-8, a transformation format of ISO 10646
• Encoding Standard – WHATWG
• TextEncoder – MDN
• Mojibake – Wikipedia