UTF-8 Encoding Explained: Bytes, Code Points and Mojibake

Sooner or later every developer meets a string that looks like café where café should be, or a database column that silently refuses an emoji, or a "limit of 100 characters" that cuts a Japanese name in half. These are not three different bugs. They are one misunderstanding about how text becomes bytes, and UTF-8 sits at the centre of it. This guide explains what UTF-8 is, how its bit patterns work with fully worked examples, why it became the default encoding of the web, and what goes wrong when two systems disagree about it. You will learn to read the bytes behind a string, recognise mojibake on sight, repair it, declare encodings correctly, and write code that counts bytes and characters without confusing them. Every code sample in this article was run, and the outputs are the real ones.

What UTF-8 is, and what it is not

Computers store numbers, not letters. To store text you need two separate agreements, and most encoding confusion comes from mixing them up.

The first agreement is a character set, a numbered list of characters. Unicode is the universal one. It assigns every character a number called a code point, written as U+ followed by hexadecimal digits. The letter A is U+0041, the euro sign is U+20AC, the Chinese character 世 is U+4E16, and the grinning face emoji is U+1F600. Code points run from U+0000 to U+10FFFF, which is 1,114,112 possible values.

The second agreement is an encoding, a rule for turning a sequence of code points into a sequence of bytes. UTF-8 is one such rule. UTF-16 and UTF-32 are others. The code point is an abstract number. The bytes are what actually sits in a file or crosses a network.

UTF-8 is a variable-length encoding. Each code point becomes one, two, three or four bytes, depending on its value:

Code point range	Bytes	Payload bits	Typical content
U+0000 to U+007F	1	7	ASCII: English letters, digits, punctuation
U+0080 to U+07FF	2	11	Latin accents, Greek, Cyrillic, Hebrew, Arabic
U+0800 to U+FFFF	3	16	Most CJK, Indic scripts, currency symbols, many symbols
U+10000 to U+10FFFF	4	21	Emoji, rare CJK, historic scripts, math alphabets

Two design goals produced this table. Common text should be compact, so the first 128 code points take a single byte. And old text should keep working, so those 128 single-byte values are identical to ASCII. A file containing only English text is byte-for-byte the same in ASCII and in UTF-8.

A short history

Early computers used 7-bit ASCII, which covers unaccented English letters, digits and punctuation. When machines spread across the world, vendors used the eighth bit to add 128 more characters, and each region picked a different set. ISO 8859-1, usually called Latin-1, covered Western European languages. Other parts of the 8859 family covered Cyrillic, Greek and more. Windows added its own variants, such as Windows-1252, which is close to Latin-1 but puts printable characters such as curly quotes and the euro sign in a range that Latin-1 reserves for control codes. East Asian languages needed thousands of characters and used multi-byte encodings that differed from vendor to vendor.

The result was chaos. A file had no reliable way to say which table it used, so a document written on one machine showed wrong characters on another. Unicode was created to give every character in every writing system one number. That solved the numbering problem, but it left the storage problem, because early proposals stored every character in two or four bytes, which doubled or quadrupled English text and broke every program that treated a zero byte as the end of a string.

UTF-8 solved both. It was designed in 1992 by Ken Thompson and Rob Pike for the Plan 9 operating system, and it was built around ASCII compatibility from the start. The current specification is RFC 3629, which limits UTF-8 to code points up to U+10FFFF and to at most four bytes per character. Earlier drafts of the format allowed sequences of five and six bytes, and those are now invalid. The Wikipedia article on UTF-8 gives a good overview of the format's history and adoption.

Today the WHATWG Encoding Standard, which governs how browsers decode text, requires UTF-8 for new web content. If you are building something new, there is rarely a reason to choose anything else.

How the bit patterns work

The entire format fits in one small table of bit patterns. Each x below is a payload bit taken from the code point:

Bytes	Byte 1	Byte 2	Byte 3	Byte 4
1	0xxxxxxx
2	110xxxxx	10xxxxxx
3	1110xxxx	10xxxxxx	10xxxxxx
4	11110xxx	10xxxxxx	10xxxxxx	10xxxxxx

There are two kinds of byte in a multi-byte sequence:

• A leading byte starts a character. Its first bits tell you how long the sequence is: 0 for one byte, 110 for two, 1110 for three, 11110 for four.
• A continuation byte always starts with 10. It carries six more payload bits.

To encode, you pick the shortest pattern whose payload capacity (7, 11, 16 or 21 bits) fits the code point, write the code point in binary, pad it with zeros on the left to fill the payload bits, and split it across the pattern from left to right.

Worked example: ASCII

The letter A is U+0041, which is 65 in decimal and 1000001 in binary. It fits in 7 bits, so it uses the one-byte pattern 0xxxxxxx. Padding to seven bits gives 1000001, and the byte is 01000001, which is 0x41. Nothing changed. This is what ASCII compatibility means in practice.

Worked example: accented Latin

The letter é is U+00E9, binary 11101001, which is eight bits and does not fit in seven. It needs the two-byte pattern 110xxxxx 10xxxxxx, which has 11 payload bits. Pad 11101001 to 11 bits: 00011101001. Split it into the first five bits 00011 and the last six bits 101001.

• Byte 1: 110 + 00011 = 11000011 = 0xC3
• Byte 2: 10 + 101001 = 10101001 = 0xA9

So é becomes the bytes C3 A9. You will see that pair constantly, because it is also the source of the most famous mojibake, as we will see below.

Worked example: CJK

The character 世 is U+4E16, which in binary is 0100 1110 0001 0110, a full 16 bits. That needs the three-byte pattern 1110xxxx 10xxxxxx 10xxxxxx, with exactly 16 payload bits. Split the 16 bits as 4, 6 and 6: 0100, 111000, 010110.

• Byte 1: 1110 + 0100 = 11100100 = 0xE4
• Byte 2: 10 + 111000 = 10111000 = 0xB8
• Byte 3: 10 + 010110 = 10010110 = 0x96

So 世 becomes E4 B8 96. The euro sign works the same way: U+20AC is 0010 0000 1010 1100, which splits into 0010, 000010, 101100 and produces E2 82 AC.

Worked example: emoji

The grinning face is U+1F600. In binary that is 1 1111 0110 0000 0000, which is 17 bits. That exceeds 16, so it needs the four-byte pattern 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx with 21 payload bits. Pad to 21 bits: 000 011111 011000 000000, split as 3, 6, 6 and 6 bits.

• Byte 1: 11110 + 000 = 11110000 = 0xF0
• Byte 2: 10 + 011111 = 10011111 = 0x9F
• Byte 3: 10 + 011000 = 10011000 = 0x98
• Byte 4: 10 + 000000 = 10000000 = 0x80

So the emoji becomes F0 9F 98 80. Here is a script that prints all five examples, so you can check the arithmetic yourself:

const enc = new TextEncoder();
const hex = (u) => [...u].map((b) => b.toString(16).padStart(2, "0")).join(" ");
const bin = (u) => [...u].map((b) => b.toString(2).padStart(8, "0")).join(" ");

for (const s of ["A", "é", "€", "世", "😀"]) {
  const u = enc.encode(s);
  const cp = s.codePointAt(0).toString(16).toUpperCase().padStart(4, "0");
  console.log(s, "U+" + cp, hex(u), "|", bin(u), "| bytes:", u.length, "length:", s.length);
}

Output from Node.js:

A U+0041 41 | 01000001 | bytes: 1 length: 1
é U+00E9 c3 a9 | 11000011 10101001 | bytes: 2 length: 1
€ U+20AC e2 82 ac | 11100010 10000010 10101100 | bytes: 3 length: 1
世 U+4E16 e4 b8 96 | 11100100 10111000 10010110 | bytes: 3 length: 1
😀 U+1F600 f0 9f 98 80 | 11110000 10011111 10011000 10000000 | bytes: 4 length: 2

Notice the last column. The emoji is one character to a human and one code point, but JavaScript reports a length of 2. We will return to that in the section on counting.

Decoding is the same table read backwards

A decoder reads one byte and looks at its top bits. If the byte starts with 0, it is a complete ASCII character. If it starts with 110, 1110 or 11110, it knows how many continuation bytes to read. It strips the marker bits from every byte, concatenates the remaining payload bits and gets the code point back. A byte that starts with 10 where a leading byte was expected is an error, and so is a leading byte that is not followed by the right number of continuation bytes.

Why UTF-8 won

Several properties make UTF-8 more than just "one of the Unicode encodings".

ASCII compatibility. Every ASCII file is a valid UTF-8 file. Tools, protocols and file formats that only care about ASCII structure, such as HTTP headers, JSON punctuation, HTML tags, XML markup and source code keywords, keep working unchanged.

No byte of a multi-byte character looks like ASCII. Every byte in a multi-byte sequence has its high bit set, so it is in the range 0x80 to 0xFF. A slash, a quote, a newline, a null byte or an angle bracket can never appear inside a multi-byte character. This is why you can search UTF-8 text for / or \n with a plain byte search and never get a false match. It is also why C code that scans for a null terminator keeps working.

Self-synchronisation. Because leading bytes and continuation bytes look different, a decoder can find character boundaries from any position. If you land in the middle of a stream, skip bytes that start with 10 until you reach one that does not, and you are at the start of a character. Corruption damages one character, not the rest of the file. Older multi-byte East Asian encodings did not have this property, and a single lost byte could scramble everything after it.

No byte-order problem. UTF-16 and UTF-32 store multi-byte units, so a reader must know whether the most significant byte comes first. UTF-8 is defined as a sequence of single bytes, so there is only one possible order and no endianness to declare.

Compact for the web. The tags and syntax of HTML, CSS, JavaScript and JSON are ASCII. A page in Japanese is mostly markup bytes (one byte each) with three bytes for each Japanese character, which is often smaller than the same page in UTF-16, where every ASCII character costs two bytes.

Sort order matches code point order. Comparing UTF-8 byte strings lexicographically gives the same result as comparing code points. UTF-16 does not guarantee that once characters outside the first 65,536 code points are involved.

These properties come with costs. Indexing the nth character is not a constant-time operation, because characters have different widths. Some scripts take more bytes in UTF-8 than in legacy encodings: a Chinese character takes three bytes in UTF-8 and usually two in older Chinese encodings. For most software the trade is worth it, and compression narrows the difference for large text.

Invalid sequences, overlong forms and U+FFFD

A decoder sees arbitrary bytes, and many byte sequences are not valid UTF-8. The standard defines exactly which ones.

What makes a sequence invalid

• A stray continuation byte. A byte starting with 10 with no leading byte before it.
• A truncated sequence. A leading byte followed by too few continuation bytes, as happens when a string is cut off mid-character.
• Bytes that never appear. The values 0xC0, 0xC1 and 0xF5 through 0xFF cannot occur in valid UTF-8.
• Surrogates. Code points U+D800 to U+DFFF are reserved for UTF-16 and are not valid scalar values, so encoding them in UTF-8 is forbidden.
• Values above U+10FFFF. Four-byte patterns that would encode a larger number are invalid.
• Overlong encodings. A code point encoded with more bytes than needed.

Overlong encodings

An overlong encoding represents a small code point using a longer pattern than necessary. The slash character is U+002F. Written correctly it is the single byte 2F. But you could write its bits into the two-byte pattern: 11 payload bits 00000101111 splits into 00000 and 101111, giving 11000000 10101111, which is C0 AF. Mathematically the bits decode to U+002F, but the standard says the shortest form is the only legal form. Decoders must reject C0 AF. That is why 0xC0 and 0xC1 never appear: any two-byte sequence starting with them would be overlong.

This rule is a security feature, not a nicety. We cover why in the security section.

What decoders do: the replacement character

When a decoder meets invalid input it has two broad choices. A strict decoder stops and reports an error. A lenient decoder substitutes the replacement character, U+FFFD, written �, and keeps going. Browsers are lenient, because showing a page with a few � symbols is better than showing nothing. Many programming APIs are strict by default and lenient on request.

Here is how the same bytes behave in JavaScript, with the default lenient decoder and with the fatal option:

const td = new TextDecoder();
console.log(JSON.stringify(td.decode(Uint8Array.of(0x63, 0xff, 0x61)))); // stray 0xFF
console.log(JSON.stringify(td.decode(Uint8Array.of(0xc0, 0xaf))));       // overlong slash
console.log(JSON.stringify(td.decode(Uint8Array.of(0xe4, 0xb8))));       // truncated 世
console.log(JSON.stringify(td.decode(Uint8Array.of(0xed, 0xa0, 0x80)))); // surrogate U+D800

try {
  new TextDecoder("utf-8", { fatal: true }).decode(Uint8Array.of(0xc0, 0xaf));
} catch (e) {
  console.log(e.name, e.message);
}

Output:

"c�a"
"��"
"�"
"���"
TypeError The encoded data was not valid for encoding utf-8

The counts of � are not arbitrary. The overlong pair C0 AF produces two replacement characters, because C0 is never valid and each byte is replaced on its own. The truncated E4 B8 produces one, because the two bytes form a valid beginning of a character that was never finished, and the standard replaces such a "maximal subpart" with a single U+FFFD. The surrogate ED A0 80 produces three, because ED can only be followed by 80 through 9F, so the second byte already breaks the sequence.

Python exposes the same choice as an errors argument. This is the part of the Python API worth memorising:

bad = b"caf\xe9"   # "café" encoded as Latin-1, then read as UTF-8

print(bad.decode("utf-8", errors="replace"))
print(bad.decode("utf-8", errors="ignore"))
print(bad.decode("utf-8", errors="backslashreplace"))
try:
    bad.decode("utf-8")
except UnicodeDecodeError as e:
    print(e)
print(bad.decode("latin-1"))

Output:

caf�
caf
caf\xe9
'utf-8' codec can't decode byte 0xe9 in position 3: unexpected end of data
café

Each handler has a place:

Handler	Effect	Use it when
strict (default)	Raises UnicodeDecodeError	You want bad data to fail loudly
replace	Substitutes U+FFFD	Display, logs, best-effort reading
ignore	Drops the bad bytes	Almost never. It silently loses data
backslashreplace	Keeps the byte as \xNN text	Diagnostics, where you must see what was wrong
surrogateescape	Maps bad bytes to lone surrogates that can round-trip	Tools that must pass through unknown bytes, such as file names

ignore deserves a warning. It turns a visible problem into an invisible one, and the text that survives can mean something different from the original. Prefer replace or backslashreplace when you cannot fail.

The same handlers exist for encoding. When a character does not exist in the target encoding, "€".encode("latin-1") raises UnicodeEncodeError, and passing errors="replace" writes a ? instead, while errors="xmlcharrefreplace" writes €. On the example "日本".encode("ascii", "xmlcharrefreplace"), Python returned b'日本', which is a valid way to put any character into pure ASCII HTML.

A decoder is a small state machine

Strict and lenient decoders both follow the same logic. The one subtle point is the allowed range of the second byte, which depends on the leading byte. E0 must be followed by A0 to BF (otherwise the result would be overlong), ED by 80 to 9F (to exclude surrogates), F0 by 90 to BF (overlong again), and F4 by 80 to 8F (to stay at or below U+10FFFF). All other second bytes are 80 to BF. These ranges come straight from the standard, and they are the reason a naive decoder that only checks the top bits is wrong.

Building an encoder and decoder by hand

Reading the bit patterns is one thing. Implementing them proves you understand them. This JavaScript module contains a complete encoder and a standards-compliant decoder that replaces invalid input with U+FFFD the same way the built-in decoder does:

// hand.js
function encode(str) {
  const out = [];
  for (const ch of str) {            // iterates code points, not UTF-16 units
    const c = ch.codePointAt(0);
    if (c < 0x80) out.push(c);
    else if (c < 0x800) out.push(0xc0 | (c >> 6), 0x80 | (c & 0x3f));
    else if (c < 0x10000) out.push(0xe0 | (c >> 12), 0x80 | ((c >> 6) & 0x3f), 0x80 | (c & 0x3f));
    else out.push(0xf0 | (c >> 18), 0x80 | ((c >> 12) & 0x3f), 0x80 | ((c >> 6) & 0x3f), 0x80 | (c & 0x3f));
  }
  return Uint8Array.from(out);
}

function decode(bytes) {
  let out = "", i = 0;
  while (i < bytes.length) {
    const b = bytes[i++];
    if (b < 0x80) { out += String.fromCharCode(b); continue; }
    let need, cp, lo = 0x80, hi = 0xbf;
    if (b >= 0xc2 && b <= 0xdf) { need = 1; cp = b & 0x1f; }
    else if (b >= 0xe0 && b <= 0xef) {
      need = 2; cp = b & 0x0f;
      if (b === 0xe0) lo = 0xa0;      // forbid overlong
      if (b === 0xed) hi = 0x9f;      // forbid surrogates
    } else if (b >= 0xf0 && b <= 0xf4) {
      need = 3; cp = b & 0x07;
      if (b === 0xf0) lo = 0x90;      // forbid overlong
      if (b === 0xf4) hi = 0x8f;      // forbid > U+10FFFF
    } else { out += "�"; continue; }
    let ok = true;
    for (let k = 0; k < need; k++) {
      const c = bytes[i];
      if (c === undefined || c < lo || c > hi) { ok = false; break; }
      cp = (cp << 6) | (c & 0x3f);
      lo = 0x80; hi = 0xbf; i++;
    }
    out += ok ? String.fromCodePoint(cp) : "�";
  }
  return out;
}
module.exports = { encode, decode };

To verify it, compare against the built-in TextEncoder and TextDecoder. The check below encodes every Unicode scalar value (all code points except the surrogates), round-trips each one, and then feeds 200,000 random byte sequences, built from a pool of tricky bytes, to both decoders:

const { encode, decode } = require("./hand.js");
const te = new TextEncoder(), td = new TextDecoder();
const same = (a, b) => a.length === b.length && a.every((x, i) => x === b[i]);

let enc = 0;
for (let cp = 0; cp <= 0x10ffff; cp++) {
  if (cp >= 0xd800 && cp <= 0xdfff) continue;
  const s = String.fromCodePoint(cp);
  if (!same(encode(s), te.encode(s)) || decode(te.encode(s)) !== s) enc++;
}

let diff = 0;
const pool = [0x41, 0xc3, 0xa9, 0xe4, 0xb8, 0xf0, 0x9f, 0x80, 0xbf, 0xed, 0xa0, 0xe0, 0xc0, 0xf4, 0x90, 0xff];
for (let k = 0; k < 200000; k++) {
  const n = 1 + Math.floor(Math.random() * 6);
  const a = Uint8Array.from({ length: n }, () => pool[Math.floor(Math.random() * pool.length)]);
  if (decode(a) !== td.decode(a)) diff++;
}
console.log("encoder mismatches:", enc, "decoder mismatches:", diff);

Output:

encoder mismatches: 0 decoder mismatches: 0

Both counts were zero. The first version of the decoder I wrote for this article did not pass the random test, which is instructive: it checked continuation bytes but not the narrower ranges after E0, ED, F0 and F4, and it replaced invalid runs with a different number of U+FFFD characters than the standard requires. Close to 900 of 20,000 random inputs disagreed with the built-in. If you ever write a decoder for production, test it this way, against a trusted implementation and with hostile input, not just with the friendly strings in your head.

You do not need to write your own in production, because every language ships a correct one. The exercise is useful because every pitfall in the rest of this article becomes obvious once you have written the bit shifts yourself.

Byte length, code points and "characters"

Most real bugs come from using one word, "length", for three different things.

• Byte length is how much storage or network the text takes.
• Code point count is how many Unicode scalar values it contains.
• Grapheme count is how many user-perceived characters it contains, the things a person would point at and call a letter.

Languages disagree about which one length returns. JavaScript strings are sequences of UTF-16 code units, so length counts those. Python 3 strings are sequences of code points, so len counts those. Neither counts bytes, and neither counts graphemes. The next script shows all three views on the same text:

const s = "café 😀";
console.log(s.length);                // UTF-16 code units
console.log([...s].length);           // code points
console.log(Buffer.byteLength(s));    // UTF-8 bytes

const family = "👨‍👩‍👧";
console.log(family.length, [...family].length,
            [...new Intl.Segmenter().segment(family)].length);

Output:

7 6 10
8 5 1

For café 😀, the string has 7 UTF-16 units (the emoji uses two), 6 code points and 10 UTF-8 bytes (é uses 2 and the emoji uses 4). The family emoji is a single visible character, but it is built from three person emoji joined by invisible zero-width joiner characters, which makes 5 code points, 8 UTF-16 units and, in Python, 18 UTF-8 bytes. The Intl.Segmenter API counts it as one grapheme.

Combining characters create the same gap. The letter é can be one code point (U+00E9) or two (e followed by the combining acute accent U+0301). They look identical on screen. Python shows the difference:

import unicodedata
a = "é"; c = "é"
print(a == c, len(a), len(c))
print(a.encode().hex(" "), "|", c.encode().hex(" "))
print(unicodedata.normalize("NFC", c) == a)

False 1 2
c3 a9 | 65 cc 81
True

The two strings are not equal, have different lengths and produce different bytes. Unicode normalisation fixes this. NFC composes characters where it can, and NFD decomposes them. If you compare, deduplicate, hash or index text, normalise it first, and pick one form for the whole system. Usernames, file names and search keys are the typical places where the difference bites.

Which length do you want?

Question	Measure	Tool
Will it fit in the column, header or packet?	Bytes	Buffer.byteLength, len(s.encode())
How many code points is it?	Code points	[...s].length, len(s) in Python
How many letters will a person see?	Graphemes	Intl.Segmenter, a grapheme library
What is the display width?	Rendering width	A layout engine. East Asian characters are often double width

Truncating without breaking characters

Cutting a byte string at an arbitrary position can land in the middle of a character. The result is a truncated sequence that decodes to U+FFFD. The test below slices the three characters of 日本語, which occupy nine bytes, at byte 4:

const b = Buffer.from("日本語");
console.log(JSON.stringify(b.subarray(0, 4).toString()));
console.log(b.subarray(0, 6).toString());

"日�"
日本

Slicing at 6 bytes works because 6 happens to be a character boundary. Slicing at 4 keeps one whole character and the first byte of the next. The fix uses self-synchronisation: step backwards from the cut point while the byte at the cut is a continuation byte.

def truncate_utf8(data: bytes, limit: int) -> bytes:
    if len(data) <= limit:
        return data
    end = limit
    while end > 0 and (data[end] & 0xC0) == 0x80:
        end -= 1
    return data[:end]

t = "日本語".encode()
for n in (4, 6, 7, 9):
    out = truncate_utf8(t, n)
    print(n, out.decode(), len(out))

4 日 3
6 日本 6
7 日本 6
9 日本語 9

The function never splits a character. A limit of 4 bytes returns the 3 bytes of 日, and a limit of 7 returns 6 bytes. Note that this prevents broken code points but not broken graphemes. A cut can still separate a base letter from its combining accent or split a family emoji into pieces. For user-facing text, truncate on grapheme boundaries and test with Intl.Segmenter in JavaScript or a grapheme-aware library in other languages.

In JavaScript there is a built-in version of the byte-aware idea. TextEncoder.encodeInto writes into a fixed buffer and reports how much it consumed:

console.log(new TextEncoder().encodeInto("日本語", new Uint8Array(4)));
// { read: 1, written: 3 }

It wrote only whole characters: one code unit was read and three bytes written, and it did not emit the first byte of the second character. This is a convenient way to fit text into a byte limit. See the MDN page for TextEncoder for the full API.

Mojibake: reading UTF-8 as something else

Mojibake, a Japanese word meaning roughly "character transformation", is the garbled text you see when bytes are decoded with the wrong encoding. The Wikipedia article on mojibake lists many variants. The bytes are fine. The decoder's assumption is wrong.

Case 1: UTF-8 read as Windows-1252 or Latin-1

This is by far the most common form. The text was correctly encoded as UTF-8, but something decoded it as a single-byte Western European encoding. Every byte becomes one character, so a two-byte character turns into two junk characters. Recall that é is C3 A9:

• In Latin-1, C3 is Ã and A9 is ©. The result is é.
• Windows-1252 gives the same pair for these two bytes.

const u = new TextEncoder().encode("café");
console.log(new TextDecoder("windows-1252").decode(u));
console.log(new TextDecoder("latin1").decode(
  new TextEncoder().encode("naïve – “quoted” €5 日本 😀")));

café
naïve – “quoted†€5 日本 😀

Look at the patterns, because they let you diagnose a problem in seconds:

• Ã followed by another symbol (é, ï, è) means two-byte Latin accents. C3 is the leading byte for U+00C0 to U+00FF.
• â€ followed by a symbol (–, “, ’) means the three-byte punctuation block. Curly quotes, dashes and ellipses start with E2 80, and E2 is â in Windows-1252 while 80 is €.
• æ, ã, ä, å and similar followed by symbols mean CJK text.
• ðŸ means a four-byte sequence beginning F0 9F, which is where most emoji live.

Notice that the Latin-1 label in the TextDecoder produced the Windows-1252 results. That is because the WHATWG standard treats the latin1 label as an alias for Windows-1252, which is what browsers do. This is a good example of why "Latin-1" in conversation is imprecise.

Case 2: double encoding

Double encoding happens when text that is already UTF-8 gets treated as Latin-1 text and converted to UTF-8 again. The first mistake produces café, and the second encodes those mistaken characters, so each of them gets bytes of its own. Python shows the stages:

m = "café".encode("utf-8").decode("cp1252")
print(m)
d = m.encode("utf-8").decode("cp1252")
print(d)

café
café

The second pass turns é into é, which is longer and more bizarre. Each extra round multiplies the garbage. A recognisable sign of double encoding is the sequence Ã or Â, which does not appear in single-pass mojibake.

Case 3: the reverse, Latin-1 read as UTF-8

If a legacy source sends café in Latin-1, the é is the single byte E9. A UTF-8 decoder sees a leading byte that announces a three-byte sequence, then finds the end of the data or a non-continuation byte. The decoder either throws or replaces it. With the replacement character you get caf�, as shown earlier. This form is lossy: the original byte is gone, replaced by a character that does not say which byte it was. That is why this direction is harder to repair than case 1.

Repairing mojibake

Case 1 and case 2 are reversible, as long as no information was lost. The procedure is mechanical: re-encode the text with the encoding that was wrongly used, then decode the resulting bytes as UTF-8.

def fix_mojibake(s, wrong="cp1252"):
    try:
        return s.encode(wrong).decode("utf-8")
    except (UnicodeEncodeError, UnicodeDecodeError):
        return s

print(fix_mojibake("café"))
print(fix_mojibake("café"))
print(fix_mojibake("café"))
print(fix_mojibake(fix_mojibake("café")))

café
café
café
café

Three details matter in that output. The function leaves already-correct text alone, because "café".encode("cp1252") yields E9, which is not valid UTF-8, so the except branch returns the input. A double-encoded string needs two passes, one for each wrong decode. And the function must be applied only when you are sure the text is garbled, because running it on healthy text can occasionally produce something else that happens to be valid.

In JavaScript, the equivalent uses the byte values of the characters directly:

const repair = (s) => new TextDecoder().decode(Uint8Array.from(Buffer.from(s, "latin1")));
console.log(repair("café"));

café

This works because Buffer.from(s, "latin1") turns each character into one byte of the same value, which restores the original UTF-8 bytes. Note that it works for characters up to U+00FF. If the mojibake contains characters such as € or ™, which Windows-1252 places in the 80 to 9F range, you need a real Windows-1252 table to map them back to bytes. Python's cp1252 codec has one.

Repair has limits:

• Undefined bytes. Windows-1252 leaves five byte values, 81, 8D, 8F, 90 and 9D, undefined. If the garbled text contains characters that came from them, or was produced by software that dropped them, the round trip fails. In Python, "Ã\x81".encode("cp1252") raises UnicodeEncodeError: 'charmap' codec can't encode character '\x81' in position 1: character maps to <undefined>.
• Replacement characters. Once � is stored, the original byte is gone. No algorithm can restore it, although you can guess from context.
• Mixed data. A column where some rows were fixed and others were not will repair some rows and wreck others. Repair row by row, test each one, and keep the original.
• Non-reversible conversions. A system that "converts" characters to ? for an unsupported encoding has destroyed the data.

If you need to inspect the bytes behind a suspicious string, paste the garbled text into the UTF-8 Encoder to see its bytes, or paste the hex bytes into the UTF-8 Decoder to see what they decode to. Comparing the two views usually reveals which encoding step went wrong.

The byte order mark

The byte order mark, or BOM, is the character U+FEFF placed at the start of a file. In UTF-16 and UTF-32 it tells the reader whether bytes are big-endian or little-endian. UTF-8 has no byte-order problem, so a UTF-8 BOM carries no ordering information. Its UTF-8 form is the three bytes EF BB BF, and its only function is to signal "this file is UTF-8".

When does that matter?

• It is allowed but optional. The Unicode standard permits a BOM in UTF-8, and does not recommend it. Many tools, especially on Unix, treat those three bytes as ordinary content.
• Some tools require it. Certain spreadsheet applications use the BOM to guess that a CSV file is UTF-8, and without it they may assume a legacy encoding and show mojibake.
• It breaks things that do not expect it. A BOM before a #! line stops a script from running. A BOM at the start of a JSON file makes some strict parsers reject it, because RFC 8259 tells implementations not to add one, and it can break byte-based comparisons. A BOM before a PHP opening tag can be output as content and break headers. A BOM at the start of a CSV file can become part of the first column's name.
• Decoders may strip it silently. Browsers' TextDecoder removes a leading BOM by default, which is why the option ignoreBOM exists.

const withBom = Uint8Array.of(0xef, 0xbb, 0xbf, 0x68, 0x69);
console.log(JSON.stringify(new TextDecoder().decode(withBom)));
console.log(JSON.stringify(new TextDecoder("utf-8", { ignoreBOM: true }).decode(withBom)));

"hi"
"hi"

The second line contains a hidden U+FEFF before the h, because ignoreBOM: true tells the decoder to keep it in the output. Python handles this with two codec names:

print("hi".encode("utf-8").hex(" "))
print("hi".encode("utf-8-sig").hex(" "))
print(b"\xef\xbb\xbfhi".decode("utf-8") == "hi")
print(b"\xef\xbb\xbfhi".decode("utf-8-sig"))

ef bb bf 68 69
ef bb bf 68 69
False
hi

Plain utf-8 leaves the BOM in the string as an invisible character, so the comparison is False. utf-8-sig writes a BOM when encoding and strips one when decoding. The practical rule: write UTF-8 without a BOM unless a specific consumer, such as a spreadsheet import, needs it, and when reading files from unknown sources, use a decoder that tolerates one.

Declaring the encoding

UTF-8 text in the wrong container is still mojibake. Bytes carry no label, so every boundary where text moves needs an explicit declaration, and every one must agree.

HTTP

The Content-Type response header can carry a charset:

Content-Type: text/html; charset=utf-8
Content-Type: application/json

For text/* types the charset matters. For JSON, the specification defines UTF-8 as the interchange encoding, so no charset parameter is needed, and adding one is harmless. Send the header from the server, and do not rely on guessing. When the header and the document disagree, browsers follow a precedence order defined in the HTML standard, and the HTTP header normally wins over a <meta> tag.

HTML

<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <title>Example</title>
  </head>

The <meta charset> declaration must be within the first 1,024 bytes of the document, so put it first in <head>, before the <title> and anything else that could contain non-ASCII text. Our guide to HTML formatting and semantic markup covers where this tag belongs in a clean document skeleton. For forms, browsers encode submissions using the page's encoding, so a page declared as UTF-8 submits UTF-8.

CSS and JavaScript

A stylesheet can start with @charset "utf-8";, but it only works as the very first thing in the file. In practice a correct Content-Type header is better. For scripts loaded with <script src>, the page's encoding is the default fallback. Modern ES modules are always decoded as UTF-8.

XML

An XML document may begin with a declaration such as <?xml version="1.0" encoding="UTF-8"?>. XML parsers default to UTF-8 when no declaration exists, and the declaration must match the real bytes. A file declared as ISO-8859-1 but saved as UTF-8 will produce mojibake or a parse error in a strict parser. See our XML formatting and structure guide for more on well-formed documents and SVG formatting and editing for the same rules in SVG files.

Databases and connections

A database has several layers, and each can be wrong independently: the server default, the database, the table, the column, and the connection used by the client. MySQL has a famous trap. Its original utf8 character set is a legacy alias for utf8mb3, which stores at most three bytes per character. It cannot hold four-byte characters, which includes all emoji and many rare CJK characters. The real UTF-8 is called utf8mb4.

Typical symptoms with the legacy set are an error such as "Incorrect string value" when inserting an emoji, or text that is silently truncated at the first four-byte character, depending on the server's strictness settings. The fix is to use utf8mb4 for the database, tables, columns and the connection, and to check the collation, which controls sorting and comparison, as well as the character set.

-- check what the connection is using
SHOW VARIABLES LIKE 'character_set%';

-- a table that can hold any Unicode character
CREATE TABLE notes (
  id BIGINT PRIMARY KEY,
  body TEXT
) CHARACTER SET utf8mb4;

Other databases handle this differently. PostgreSQL's UTF8 encoding is full UTF-8, and it rejects some invalid input such as a null character in text. In every database, remember the connection encoding: a client library that sends Latin-1 into a UTF-8 column will either fail or store garbled bytes that look fine until another client reads them. Our SQL formatting guide covers readable queries, and the same care applies to the character set clauses in your schema.

Note that in many databases a length declared as VARCHAR(255) is counted in characters, not bytes, but index key limits are often counted in bytes. A utf8mb4 column can therefore need up to four times as many bytes for an index as an ASCII column of the same declared length.

Files and the command line

Plain text files have no declaration at all. Your editor, language runtime and operating system each have defaults. In Python 3, open files with an explicit encoding rather than relying on the platform default:

with open("notes.txt", "w", encoding="utf-8") as f:
    f.write("café €5 世 😀\n")

In Node.js, fs.readFileSync(path, "utf8") and fs.writeFileSync with a string use UTF-8 by default. On a terminal, the locale settings decide how bytes are displayed, so a UTF-8 file can look broken on a terminal configured for another encoding even though the file itself is fine.

Real-world use cases

Understanding UTF-8 pays off in everyday work.

Web pages and APIs. Every page and JSON response should be UTF-8 end to end. Doing so allows names, addresses and messages in any language. Our guide to JSON formatting discusses the format, and the payload size article shows why letting non-ASCII text through as real UTF-8 rather than escape sequences saves bytes.

Databases and data migrations. A migration from a legacy character set to utf8mb4 has to convert data, not just change a label. Changing only the declared charset of a column that holds UTF-8 bytes already mislabelled as Latin-1 is how double encoding gets baked into a database.

CSV and spreadsheet exports. Exported files that open as mojibake in a spreadsheet usually lack a BOM or lack a declared encoding in the import dialog.

Email and messaging. Message headers and bodies declare their charset separately, and mixed or missing declarations are a classic cause of garbled subjects.

Network protocols and binary formats. Length-prefixed strings in a protocol normally state the length in bytes. A client that computes the prefix from a UTF-16 length will produce a wrong value as soon as a non-ASCII character appears.

URLs. Non-ASCII characters in a URL are sent as the UTF-8 bytes of the character, with each byte percent-encoded. For example, é becomes %C3%A9. That is why you see %C3%A9 in links, and it is the same pair C3 A9 from our worked example.

Logs and debugging. Binary-safe logging shows bad bytes as \xNN, which is the backslashreplace behaviour. It is far more useful than a row of �.

Internationalised identifiers. File names, usernames and email addresses can contain any characters, so they must be normalised and compared consistently.

Security considerations

Encoding bugs are not just cosmetic. They have a long history as a way around input checks.

Overlong encodings and filter bypass

Recall that C0 AF is an overlong encoding of /. Suppose a web server rejects any path containing ../ to prevent directory traversal. If the check works on raw bytes, and a lower layer later decodes with a lenient decoder that accepts overlong forms, then ..%C0%AF passes the check and decodes to ../. Early web servers and other software had exactly this class of vulnerability. The principle is general: the check and the use must see the same data in the same form.

This is why RFC 3629 insists that decoders reject non-shortest forms, and why a decoder you write yourself must be tested against these cases. It is also why validation belongs after decoding, on the canonical form, not before.

Normalisation and look-alike characters

Different code point sequences can look identical, as with the two forms of é. Some Unicode characters also resemble Latin letters, such as a Cyrillic а next to a Latin a. Consequences:

• Duplicate accounts. Two usernames that look the same but differ in code points can both be registered. Normalise and consider rejecting mixed-script names.
• Spoofed domains and file names. Look-alike characters in a URL or an executable's name can trick users.
• Bypassing keyword filters. A filter searching for a banned word will miss a version containing a combining mark or a zero-width character.
• Case mapping surprises. Uppercasing or lowercasing some characters changes their length, and comparisons done before and after case conversion can differ.

Apply normalisation (usually NFC, or NFKC for identifiers) once at the boundary, then validate and store the normalised value.

Validate at the edge

• Decode strictly at trust boundaries. Use strict decoding and reject invalid sequences, rather than replacing them and moving on. In JavaScript pass fatal: true. In Python leave the default strict.
• Reject or strip control characters that your application does not need, including null bytes, which can truncate strings in C-based libraries.
• Check length in the unit that matters. If your storage limit is in bytes, measure bytes, or an attacker will pick characters that expand.
• Escape on output for the target context. UTF-8 does not make text safe. HTML, SQL and shell contexts each need their own escaping.
• Be careful with bidirectional control characters. Text that includes right-to-left override characters can display differently from its logical order, which has been used to disguise file names and source code.

Truncation and cut-off sequences

Truncating a string mid-character can leave a partial sequence that downstream code mishandles. When the truncated text is then fed to a decoder that swallows following bytes, for example by treating a leading byte as consuming the next characters, it can alter neighbouring content. Standards-compliant decoders handle this correctly, but custom parsers often do not. Truncate on character boundaries, as in the earlier example.

Performance considerations

UTF-8 is fast in practice, and a few facts help you reason about cost.

ASCII fast paths. Because most real-world text contains long runs of ASCII bytes, decoders and string libraries optimise for them. A run of bytes below 0x80 can be copied or validated in bulk.

Indexing is not O(1). To find the nth character you must scan from the start, unless you keep an index. Languages that store UTF-8 internally, or that expose byte offsets, make you work with offsets rather than positions. If your code does s[i] in a loop over a large UTF-8 string, check what it costs.

Size. For text in Latin scripts with occasional accents, UTF-8 is close to one byte per character. For Greek, Cyrillic, Hebrew and Arabic, it is about two bytes per letter. For Chinese, Japanese and Korean, it is three bytes per character, which can be larger than a legacy two-byte encoding but is usually reduced sharply by compression. HTTP compression with gzip or Brotli removes much of the difference, as discussed in the payload size guide.

Validation cost. Strict validation costs a little extra per byte compared with copying. It is the right trade at input boundaries and unnecessary for data you already validated.

Conversions between UTF-16 and UTF-8. JavaScript engines and many platforms use UTF-16 internally, while files and networks use UTF-8. Each boundary crossing is a conversion and a copy. Keep text as strings inside your program, encode once on the way out, decode once on the way in, and avoid encoding and decoding repeatedly in loops.

A quick measurement with the same method from the payload article shows the size effect on a sample you can reproduce. The string café €5 世 😀 has 11 code points, and Python's len(s.encode("utf-8")) returned 19 bytes for it. That is a small sample, so treat it as an illustration of the ratio, not a benchmark.

Accessibility and internationalisation

Correct encoding is an accessibility feature. Screen readers and text-to-speech engines receive characters, not bytes, so mojibake gets read aloud as gibberish or as a stream of unrelated symbols. A page with a wrong encoding is unreadable to everyone, but it hits hardest the users whose language needs the characters that break first.

• Declare the language with lang attributes alongside the encoding, so assistive technology chooses the right pronunciation rules.
• Do not replace text with images to dodge encoding problems. Fix the encoding instead.
• Keep names intact. Users with accented or non-Latin names notice when software mangles their name, and many people experience that as a lack of respect for them.
• Test with real scripts. Include Arabic or Hebrew (right-to-left), Chinese, Japanese or Korean (wide characters), and combining marks (stacked diacritics) in your test data, not only accented European names.
• Allow enough space. Text in other languages can be longer or taller than English text, and a layout built around ASCII widths may truncate or overlap.

Step-by-step: inspect and convert with DevUtilX

When you need a quick answer, such as "what bytes is this character?" or "what does this hex decode to?", browser tools beat writing a script. DevUtilX provides two that process your input locally in your browser, so nothing you paste is uploaded.

Encode text to UTF-8 bytes

1. Open the UTF-8 Encoder.
2. Paste or type your text, for example café 😀.
3. Run the conversion and read the resulting bytes. For this text you should see 63 61 66 c3 a9 20 f0 9f 98 80, which is ten bytes for six characters.
4. Compare the byte count to the character count. If the bytes exceed a limit in your database or protocol, you have found your truncation problem.
5. Copy the bytes in the format you need, such as hex for a test fixture or percent-encoded for a URL.

Decode bytes back to text

1. Open the UTF-8 Decoder.
2. Paste the bytes, for example e4 b8 96 e7 95 8c.
3. Run the decoder and read the text. Here it is 世界.
4. If the output contains �, the input has an invalid or truncated sequence. Look at where it appears, since the replacement character marks the position where decoding went wrong.
5. Try a deliberately damaged input, such as e4 b8, to see how a lenient decoder reports a truncated character.

Diagnose mojibake with both tools

If you have a garbled string such as café:

1. Reason about the pattern. é suggests that C3 A9 was read as two single-byte characters.
2. Put C3 A9 into the decoder. It returns é, which confirms the hypothesis.
3. In your pipeline, find the step that decodes with the wrong encoding, and fix the declaration there, not the symptom.

Because these tools run in your browser, they are safe for text that contains names or internal identifiers, but as with any tool, avoid pasting passwords or tokens.

Common mistakes

• Guessing instead of declaring. Leaving out charset and hoping the browser or tool guesses right. Declare UTF-8 explicitly at every layer.
• Declaring one thing and saving another. A <meta charset="utf-8"> tag on a file saved by an editor as Windows-1252 yields mojibake on every accent.
• Changing the label instead of converting the data. Relabelling a column or file does not change its bytes. Convert, then label.
• Using length as a byte count. Limits in bytes need Buffer.byteLength or len(s.encode()).
• Using length as a character count. In JavaScript an emoji counts as 2. A count of what people see needs a grapheme segmenter.
• Slicing bytes or UTF-16 units in the middle of a character. Use a boundary-aware truncation, as shown above.
• Using MySQL utf8. It is three bytes only. Use utf8mb4 and verify the connection.
• Using errors="ignore". It hides data loss. Prefer replace or backslashreplace and log the damage.
• Decoding with a lenient decoder and then validating. Validate in strict mode, or validate the raw bytes before use.
• Double decoding or double encoding. Each layer that "helps" by converting can make things worse. Know which layer owns the conversion.
• Comparing without normalising. é and e plus U+0301 are different strings that render the same.
• Writing a BOM everywhere. It breaks shell scripts, JSON consumers and CSV headers in some tools. Write it only when a specific consumer needs it.
• Mixing encodings in a single file. A text file concatenated from sources with different encodings cannot be decoded correctly, since no single decoder fits. Convert each source first.

Troubleshooting: symptom, cause, fix

Symptom: é appears as é, and ’ appears as ’

Cause. UTF-8 bytes were decoded as Windows-1252 or Latin-1. The most common culprits are a missing or wrong Content-Type charset, a missing <meta charset>, or a database connection using a legacy charset.
Fix. Declare UTF-8 in the HTTP header and in the document, and set the connection charset. To repair stored data, re-encode with the wrong encoding and decode as UTF-8, as in the earlier example.

Symptom: text looks like café

Cause. Double encoding. UTF-8 text was converted from "Latin-1" to UTF-8 a second time.
Fix. Find the extra conversion step, remove it, and repair existing data by applying the reverse conversion once per extra pass.

Symptom: caf� or a row of � characters

Cause. A decoder expected UTF-8 and received bytes that are not valid UTF-8, typically text in Latin-1 or another legacy encoding. It may also be truncated input.
Fix. Find the true source encoding and decode with that, then re-encode as UTF-8. If the data was already stored with �, the original bytes are lost and need to be restored from the source.

Symptom: emoji turns into ???? or an insert fails with "Incorrect string value"

Cause. The column or connection uses a character set that cannot represent four-byte characters, such as MySQL utf8 (three bytes), or an encoding like Latin-1 that cannot represent the character at all.
Fix. Switch to utf8mb4 for the database, tables, columns and connection, then convert existing data. Verify with SHOW VARIABLES LIKE 'character_set%'.

Symptom: the first column header in a CSV has strange characters before the name

Cause. The file starts with a UTF-8 BOM that the reader treats as part of the first field.
Fix. Read with utf-8-sig in Python, or strip a leading U+FEFF in your parser. When exporting, write a BOM only for consumers that need it.

Symptom: a script fails with a syntax error on its first line

Cause. A BOM before a shebang line, or before a language directive.
Fix. Save the file as UTF-8 without BOM.

Symptom: a string cut to "100 characters" is shorter, or ends in �

Cause. The limit was applied to bytes or UTF-16 units, and the cut landed inside a multi-byte or surrogate sequence.
Fix. Truncate on code point or grapheme boundaries, or step back past continuation bytes as in truncate_utf8. Choose the limit's unit deliberately.

Symptom: two usernames look identical but both register

Cause. They use different code point sequences, for example precomposed é and e plus a combining accent, or look-alike letters from other scripts.
Fix. Normalise to NFC or NFKC at input, store the normalised form, and consider script-mixing checks for identifiers.

Symptom: a JSON or API payload shows é instead of é

Cause. The serialiser escapes non-ASCII characters. This is valid JSON but looks odd and makes payloads larger.
Fix. If you control the producer, turn off the escaping, for example ensure_ascii=False in Python, and make sure the response is declared as UTF-8. If you only consume the data, the parser decodes the escapes for you.

Symptom: a decode error says "invalid start byte" or "unexpected end of data"

Cause. "Invalid start byte" means a byte that cannot begin a character appeared, such as 0xE9 followed by an ASCII letter in Latin-1 text, or 0xFF. "Unexpected end of data" means a sequence was cut short.
Fix. Look at the bytes at the reported position. A single high byte surrounded by ASCII points to a legacy single-byte encoding. A cut at the end of a chunk points to a stream that was split between characters. In that case, use an incremental decoder, such as TextDecoder with { stream: true } or Python's codecs.getincrementaldecoder("utf-8"), which keeps partial characters between chunks.

Symptom: a file works on one machine and fails on another

Cause. The runtime or editor uses a different default encoding on each platform, and the code did not specify one.
Fix. Always pass an explicit encoding when reading and writing text files, and set the locale in scripts and containers.

Sorting and comparison, briefly

Byte-wise comparison of UTF-8 strings matches code point order, which is stable and fast, and is the right choice for keys, indexes and anything that must be deterministic. It is not the order a human expects. In byte order, uppercase letters sort before lowercase, and accented letters sort after z. Python's output shows it:

print(sorted(["z", "é", "a", "e"]))

['a', 'e', 'z', 'é']

Sorting for people needs locale-aware collation, which orders é next to e and follows the conventions of a language. In JavaScript use Intl.Collator or localeCompare. In databases, pick a collation that matches your audience. Keep the distinction clear: use code point order for machines, and collation for people. Also remember that case-insensitive comparison and equality in a database depend on the collation too, which is another reason to set it explicitly along with the charset.

Best practices checklist

• Use UTF-8 everywhere for new systems: files, databases, APIs, templates and message formats.
• Declare it at every boundary: HTTP Content-Type, <meta charset="utf-8"> early in <head>, XML declarations, file open calls and database connections.
• Decode once at input and encode once at output. Use text strings in between.
• Separate bytes from text in your types and variable names.
• Decode strictly at trust boundaries and reject invalid input, and be deliberate when you choose a lenient mode.
• Normalise text to one form before comparing, deduplicating or indexing.
• Measure the right length: bytes for storage and protocols, code points or graphemes for display and user-facing limits.
• Truncate on boundaries, never in the middle of a character.
• Use utf8mb4 in MySQL and check the connection and collation as well.
• Skip the BOM unless a consumer needs it, and read files with a BOM-tolerant decoder.
• Test with hostile data: emoji, combining marks, right-to-left text, CJK and invalid byte sequences.
• Keep the original bytes when repairing mojibake, so you can retry.
• Use streaming decoders for chunked input so characters are never split between chunks.
• Log with escapes such as backslashreplace when the data may be invalid.

Comparison: Unicode encodings and legacy alternatives

Encoding	Bytes per character	ASCII compatible	Notes
UTF-8	1 to 4	Yes	Default for the web, files and most APIs. No byte-order issue
UTF-16	2 or 4	No	Used internally by JavaScript, Java and Windows APIs. Needs endianness handling
UTF-32	4	No	Fixed width, simple indexing, wasteful in storage
ASCII	1	Yes (it is ASCII)	Only 128 characters, no accents
ISO 8859-1 (Latin-1)	1	Yes	Western European only. Every byte value maps to a character
Windows-1252	1	Yes	Latin-1 variant with extra punctuation in the 80 to 9F range. Often mislabelled as Latin-1
Shift_JIS, GBK, Big5	1 or 2	Mostly	Legacy East Asian encodings, still encountered in old data
UTF-8 with BOM	1 to 4	Yes, apart from the mark	Same as UTF-8 with a three-byte signature. Needed by some spreadsheets, harmful elsewhere

UTF-16 deserves a sentence of context because it explains the JavaScript surprises above. It encodes code points up to U+FFFF as one 16-bit unit and larger ones as a pair of units called a surrogate pair. That is why the emoji U+1F600 has a JavaScript length of 2 and why lone surrogate values cannot be represented in UTF-8.

FAQ

Is UTF-8 the same as Unicode?
No. Unicode is the catalogue of characters and their code points. UTF-8 is one way to turn those code points into bytes. UTF-16 and UTF-32 are others.

How many bytes does a UTF-8 character use?
One to four. ASCII takes one, most accented Latin, Greek, Cyrillic, Hebrew and Arabic letters take two, most CJK characters take three, and emoji take four.

Why does my emoji have a length of 2 in JavaScript?
JavaScript strings are sequences of UTF-16 code units, and emoji outside the first 65,536 code points need two of them. Use the spread operator or Array.from to count code points, and Intl.Segmenter to count user-perceived characters.

What is the replacement character?
U+FFFD, shown as �. Decoders insert it when they meet bytes that are not valid in the encoding. Once it is stored, the original bytes are gone.

What does é mean?
It is the typical sign of UTF-8 text decoded as Windows-1252 or Latin-1. The two bytes C3 A9 of é were read as two separate characters.

Should I add a BOM to UTF-8 files?
Usually not. UTF-8 has no byte-order ambiguity, and the BOM breaks shell scripts and some JSON and CSV consumers. Add it only if a particular tool, such as an older spreadsheet import, needs it to detect UTF-8.

What is the difference between MySQL utf8 and utf8mb4?
MySQL's utf8 stores at most three bytes per character, so it cannot hold emoji or other four-byte characters. utf8mb4 is complete UTF-8. Use it for new databases and migrate old ones.

Is it safe to decode with errors="ignore"?
Rarely. It deletes invalid bytes without any sign that data was lost. Use strict decoding for validation and replace or backslashreplace when you must keep going.

What is an overlong encoding, and why is it dangerous?
It is a longer-than-necessary byte sequence for a code point, such as C0 AF for /. The standard forbids it. If one layer checks the bytes and another layer accepts the overlong form, the check can be bypassed, which is the basis of historic path traversal attacks.

How do I find out what encoding a file uses?
You cannot be certain, because bytes carry no label. Check for a BOM, check whether the bytes are valid UTF-8 (random Latin-1 text with accents is almost never valid UTF-8), and look at metadata such as an HTTP header. Detection libraries make educated guesses, but they remain guesses.

Why do é as one character and é as two characters compare as different?
One is the precomposed code point U+00E9 and the other is e followed by a combining accent, U+0065 U+0301. Normalise both to NFC before comparing.

Does UTF-8 take more space than UTF-16?
For mostly ASCII text, UTF-8 is about half the size. For text that is mostly CJK, UTF-8 uses three bytes per character against two in UTF-16, so UTF-16 can be smaller before compression. After compression the difference is usually small.

Can I repair mojibake after the fact?
Often yes, if the bytes were not lost. If the text contains é-style patterns, re-encode it with the wrongly used encoding and decode as UTF-8. If it contains �, the information is gone and you need the original source.

Does JSON have to be UTF-8?
RFC 8259 requires UTF-8 for JSON exchanged between systems. Parsers may also accept \uXXXX escapes for any character, so even pure ASCII JSON can carry any Unicode text.

Conclusion

UTF-8 is simple once you see its shape. One to four bytes per code point, leading bytes that announce the length, continuation bytes that begin with 10, and ASCII untouched. Almost every encoding problem you will meet comes from a mismatch at one boundary: a missing declaration, a legacy database charset, a length counted in the wrong unit, or a conversion applied twice. Fix the declaration, decode and encode once, count the right thing, and the garbled text goes away.

To try the ideas on your own strings, open the UTF-8 Encoder to see exactly which bytes a piece of text becomes, and the UTF-8 Decoder to turn bytes back into text and spot invalid sequences. Both run locally in your browser. For related reading, see our guides to JSON formatting and JSON minification and payload size, which cover how encoding choices affect the data you send over the wire.

Further reading

• UTF-8 – Wikipedia
• RFC 3629: UTF-8, a transformation format of ISO 10646
• TextEncoder – MDN
• TextDecoder – MDN
• Unicode – Wikipedia
• Mojibake – Wikipedia
• Encoding Standard – WHATWG
• Unicode HOWTO – Python documentation