Unicode Code Points, Escapes and Text-to-Unicode Conversion

You paste a username into a form and the site says it is 11 characters long, though you typed one emoji. A file name looks identical to another, yet the system treats them as different files. A search for "cafe" fails to find "café" even though both are on the page. A string reversed in JavaScript comes back with a broken question-mark box in the middle. All four bugs have the same root: text is not a sequence of letters, it is a sequence of numbers that software interprets in layers, and each layer has its own idea of what a "character" is. This guide is for developers, testers and anyone who has stared at a strange symbol and wondered what it really was. You will learn what Unicode code points are, how they differ from bytes and from what people see on screen, how escapes work in JavaScript, Python, HTML and CSS, how to convert text to code points and back, and how to avoid the security and troubleshooting traps that come with international text.

What Unicode is, and what it is not

Unicode is a character set: a giant catalogue that assigns a unique number, called a code point, to every character it covers. The capital letter A is 65. The Greek letter lambda is 955. The grinning face emoji is 128512. Unicode does not say how those numbers are stored in a file or sent over a network. That job belongs to an encoding such as UTF-8, UTF-16 or UTF-32.

Keeping the two ideas apart removes most of the confusion in this topic.

• Character set (Unicode): a mapping from characters to integers. "Which number is the euro sign?" Answer: 8364, written U+20AC.
• Encoding (UTF-8, UTF-16, UTF-32): a rule for turning those integers into bytes. "How do I store 8364 in a file?" Answer depends on the encoding.
• Glyph (font): the picture drawn on screen. One code point can have many glyph shapes, and several code points can combine into one picture.

The standard is maintained by the Unicode Consortium, whose home is home.unicode.org. The Consortium publishes the code point assignments, character names, properties, and the algorithms for things such as normalization and bidirectional text.

A short history: from 7-bit ASCII to a universal catalogue

Early computers used tiny character sets. ASCII, standardised in the 1960s, used 7 bits and covered 128 values: English letters, digits, punctuation and control codes. That was enough for American English and little else. Vendors then used the spare eighth bit to add 128 more characters, and every region made a different choice. Western Europe got one table, Russia another, Greece a third. The same byte value meant different letters depending on the table in use, and opening a file with the wrong table produced garbled text. East Asian languages needed thousands of characters and used multi-byte schemes that were incompatible with one another.

Unicode set out to end this by giving every character in every script one permanent number. The first versions used 16 bits, which seemed like enough room. It was not. The standard grew past 65,536 characters, and the design was extended to a larger code space, which is where surrogate pairs, discussed below, come from.

Two features of the design are worth remembering. First, the first 128 code points are identical to ASCII, and the next 128 match the common Latin-1 table, so old text maps cleanly into the new system. Second, code points are never reassigned. Once a character receives a number, that number belongs to it forever, which is why a document written decades ago still means the same thing today. New versions of the standard add characters, including new emoji, but do not move existing ones.

Code point notation: U+XXXX

Unicode writes code points as "U+" followed by hexadecimal digits, with at least four digits. So the letter A is U+0041, the euro sign is U+20AC, and the grinning face is U+1F600. Larger values use five or six digits and never carry leading zeros beyond the minimum of four.

The code space runs from U+0000 to U+10FFFF, which is 1,114,112 possible values (17 planes of 65,536 each). Not all of them are characters. Some are reserved, some are permanently unassigned "noncharacters", and a block in the middle is reserved for surrogates, which can never be valid characters on their own.

Here is how the notation compares to the representations you meet in code. All values below come from running the scripts in this article.

Character	Code point	Decimal	UTF-8 bytes	UTF-16 code units
A	U+0041	65	41	0041
é	U+00E9	233	C3 A9	00E9
€	U+20AC	8364	E2 82 AC	20AC
世	U+4E16	19990	E4 B8 96	4E16
😀	U+1F600	128512	F0 9F 98 80	D83D DE00

The UTF-8 lengths for A, é, € and 😀 measured in Node.js were 1, 2, 3 and 4 bytes. That variable width is the defining trait of UTF-8: ASCII stays one byte, which makes it compatible with older tools, and everything else takes two to four.

Planes, blocks and categories

The code space is divided into 17 planes. You only need to know a few of them.

Plane	Range	Name	What lives there
0	U+0000 to U+FFFF	Basic Multilingual Plane (BMP)	Most modern scripts, common symbols, CJK core
1	U+10000 to U+1FFFF	Supplementary Multilingual Plane	Historic scripts, most emoji, math alphanumerics
2	U+20000 to U+2FFFF	Supplementary Ideographic Plane	Rare CJK ideographs
14	U+E0000 to U+EFFFF	Supplementary Special-purpose Plane	Tag characters, variation selectors
15 and 16		Private use planes	Characters with no standard meaning

Everything outside plane 0 is called "supplementary". These are exactly the characters that trip up systems built when 16 bits looked like enough. A string function that is fine with "café" can fail on a poem containing a single emoji.

Inside planes, characters are grouped into named blocks, such as Basic Latin, Cyrillic or Emoticons. A block is a convenience for people reading charts. It is not a reliable way to classify a character, because blocks have gaps and characters sometimes sit in a different block than you would guess. For real classification, use properties.

General categories

Every code point has a general category, a two-letter code that says what kind of character it is. This is far more useful for programs than blocks. Python's unicodedata.category returned these values in my run:

Character	Category	Meaning
A	Lu	Letter, uppercase
é	Ll	Letter, lowercase
combining acute accent (U+0301)	Mn	Mark, nonspacing
😀	So	Symbol, other
zero width space (U+200B)	Cf	Other, format
5	Nd	Number, decimal digit
space	Zs	Separator, space

Categories let you write correct validation. A regular expression that accepts "letters" should not list ranges like [a-zA-Z], which rejects every non-English name. In JavaScript, the Unicode property escape \p{L} matches letters in any script when the u flag is on. In my test, /^\p{L}+$/u.test("héllo") returned true. The same idea applies to marks (\p{M}), digits (\p{Nd}) and so on.

Code points, code units and graphemes: three meanings of "length"

This is the heart of most string bugs. A piece of text can be measured at three levels, and each gives a different number.

1. Code units are the fixed-size chunks an encoding uses. In UTF-16 a unit is 16 bits. In UTF-8 it is one byte.
2. Code points are the Unicode numbers. One code point may need one or several code units.
3. Grapheme clusters are what a reader perceives as one character. One cluster may contain several code points.

JavaScript strings are sequences of UTF-16 code units, so the length property counts units. Python 3 strings are sequences of code points, so len counts code points. Neither counts what a person sees. Here is a Node.js script that shows all three measures on five strings, using Intl.Segmenter for graphemes.

const seg = new Intl.Segmenter("en", { granularity: "grapheme" });
const samples = {
  family: "👨‍👩‍👧‍👦",
  flag: "🇯🇵",
  thumbsUp: "👍🏽",
  eAcute: "é",
};
for (const [name, text] of Object.entries(samples)) {
  console.log(name, {
    units: text.length,
    codePoints: [...text].length,
    graphemes: [...seg.segment(text)].length,
    hex: [...text].map((c) => c.codePointAt(0).toString(16)),
  });
}

The output, measured on Node.js 22:

String	UTF-16 units	Code points	Graphemes	Code points in hex
Family emoji	11	7	1	1f468, 200d, 1f469, 200d, 1f467, 200d, 1f466
Flag of Japan	4	2	1	1f1ef, 1f1f5
Thumbs up with skin tone	4	2	1	1f44d, 1f3fd
e plus combining acute	2	2	1	65, 301

A single family emoji is eleven code units long in JavaScript. That is the 11-character username from the intro.

Surrogate pairs

UTF-16 stores BMP code points as one 16-bit unit. Supplementary code points do not fit, so UTF-16 splits them into two units called a surrogate pair. The first, the high surrogate, is in the range D800 to DBFF. The second, the low surrogate, is in DC00 to DFFF. The math is simple: subtract 0x10000 from the code point, put the top 10 bits into the high surrogate offset from 0xD800, and the bottom 10 bits into the low surrogate offset from 0xDC00.

For U+1F600 the calculation gives 0xD83D and 0xDE00, and my script confirmed both values. The Python encoder agrees: "😀".encode("utf-16-le").hex() returned 3dd800de, which is the same two units written in little-endian byte order.

Because the surrogate range is reserved for this purpose, a lone surrogate is invalid text. JavaScript still lets you create one, since its strings are just arrays of units. That is where the broken box comes from.

Counting and slicing in JavaScript

The string "A😀é" has three characters in any human sense. JavaScript reports otherwise:

const s = "A😀é";
console.log(s.length);              // 4
console.log(Array.from(s).length);  // 3
console.log(s.codePointAt(1).toString(16));  // 1f600
console.log(s.charCodeAt(1).toString(16));   // d83d
console.log(s.charCodeAt(2).toString(16));   // de00
console.log(s.split("").reverse().join(""));  // "é\ude00\ud83dA"
console.log([...s].reverse().join(""));       // "é😀A"

split("") cuts between code units and tears the emoji in half. Reversing then puts the halves in the wrong order, and the result contains two lone surrogates. Spreading the string with [...s] or calling Array.from(s) uses the string iterator, which walks by code point and keeps pairs together.

Be careful, though. Code-point iteration fixes surrogate pairs but not graphemes. Reversing "é" by code point puts the accent in front of the e, and the accent attaches to whatever comes before it, which is now nothing or a different letter. For user-facing text, segment by grapheme.

You can also detect a damaged string. Newer JavaScript engines provide isWellFormed() and toWellFormed(). In my run, "😀".slice(0,1).isWellFormed() returned false, and "a\uD83Db".toWellFormed() returned "a�b", replacing the lone surrogate with the replacement character. Check your target runtimes before relying on these methods in code that must run in older environments.

Combining marks and normalization

Many characters can be written in more than one way. The letter é is a single code point, U+00E9. It can also be written as a plain e (U+0065) followed by a combining acute accent (U+0301). Both render identically, and both are correct Unicode. They are not equal as strings.

const a = "é";
const b = "é";
console.log(a === b);                       // false
console.log(a.length, b.length);            // 1 2
console.log(a.normalize() === b.normalize());  // true

This is not a rare edge case. macOS file systems have historically stored file names in decomposed form, while many web forms and Windows applications produce composed text. A user who types an accented name in one place and searches for it in another can silently miss the match.

The four normalization forms

Unicode defines normalization forms that convert text to a canonical shape, so equivalent strings become identical.

Form	Name	What it does
NFC	Canonical Composition	Decomposes, then recomposes where possible. Shortest common form.
NFD	Canonical Decomposition	Splits characters into base plus combining marks.
NFKC	Compatibility Composition	Also replaces compatibility variants with plain equivalents, then composes.
NFKD	Compatibility Decomposition	Compatibility replacements, then full decomposition.

The "K" forms are lossy. They treat visually distinct characters as equivalent when they are close in meaning. In Python I ran the string "fié²" (the fi ligature, e-acute, superscript two) through all four forms and recorded the code points:

Form	Resulting code points
NFC	fb01, e9, b2
NFD	fb01, 65, 301, b2
NFKC	66, 69, e9, 32
NFKD	66, 69, 65, 301, 32

NFC and NFD kept the ligature and the superscript, because they are different characters with their own meaning. NFKC and NFKD expanded the ligature into the two letters f and i, and turned the superscript two into a plain 2. In JavaScript, "Ａ".normalize("NFKC") (a fullwidth A) gave "A", "①".normalize("NFKC") gave "1", and "x²".normalize("NFKC") gave "x2".

Which form should you use?

• NFC for storing and comparing text. It is the most widely recommended default and is what the web generally expects.
• NFD when you want to process accents separately, for example to strip them for search indexing.
• NFKC for identifiers and search keys, where you want "fi" and "fi" to match. Do not use it on user content you must preserve, because it destroys information such as the meaning of x² as an exponent.

A common recipe for accent-insensitive search is to decompose and then remove combining marks:

const strip = (s) => s.normalize("NFD").replace(/\p{M}/gu, "");
console.log(strip("café Ångström"));  // "cafe Angstrom"

Note that this recipe works for Latin accents and fails for scripts where marks are essential. Removing combining marks from Hindi or Arabic text, for example, changes the words. Restrict the technique to the scripts you intend it for.

Singletons and surprises

Normalization sometimes changes a character to a different code point even when nothing is composed. The Angstrom sign U+212B is a distinct code point that normalizes to the ordinary Å, U+00C5. My test confirmed it: "Å".normalize("NFC") produced code point c5. Text that looks the same can therefore differ at the code point level in ways that only normalization reveals.

Grapheme clusters, emoji and zero width joiners

What a reader calls a character is formally an extended grapheme cluster. The Unicode text segmentation rules define where one ends and the next begins. A cluster usually has a base character followed by zero or more extending characters, but emoji push the idea much further.

Combining sequences

An accent or a Hindi vowel sign is a combining or extending character. It has no meaning alone and attaches to the character before it. These are the "Mn" and "Mc" categories you saw above. Python reports the combining class of U+0301 as 230, which describes where it is drawn relative to the base. Normalization uses those classes to put multiple marks on one base into a predictable order.

Regional indicators and flags

A flag emoji is not one code point. It is two regional indicator symbols that spell a country code. The flag of Japan above was U+1F1EF followed by U+1F1F5, the indicator versions of the letters J and P. A renderer that knows the pair draws a flag. One that does not shows two boxed letters. The takeaway is practical: you cannot slice a string between the two halves of a flag without changing its meaning.

Skin tones and modifiers

The thumbs-up with a medium skin tone is U+1F44D followed by U+1F3FD, a modifier code point. A renderer combines them into one picture. If the modifier is dropped, you get the default yellow hand, so trimming a string by code point can silently change who an emoji represents.

Zero width joiner sequences

The family emoji is a chain: man, joiner, woman, joiner, girl, joiner, boy. The joiner is U+200D, the zero width joiner (ZWJ), an invisible character that asks the renderer to fuse neighbours into a single glyph if a font supports the combination. That is why the family emoji measured seven code points and eleven UTF-16 units but one grapheme. On a platform without a font for that sequence, you see the four faces side by side. The text is still the same seven code points, which is why copying the sequence between systems always preserves meaning even when rendering differs.

Segmenting by grapheme

Use Intl.Segmenter in JavaScript when the answer must match what a person sees, such as a character counter or a truncation function:

const seg = new Intl.Segmenter(undefined, { granularity: "grapheme" });
const truncate = (s, n) => [...seg.segment(s)].slice(0, n).map((x) => x.segment).join("");
console.log(truncate("ab👨‍👩‍👧‍👦cd", 3));  // ab👨‍👩‍👧‍👦
console.log("ab👨‍👩‍👧‍👦cd".slice(0, 3));    // "ab" plus half an emoji

In my run the first call kept the whole family emoji, and the second returned ab followed by a lone surrogate, which displays as a replacement box. Intl.Segmenter is available in current Node.js releases (the sandbox used version 22) and in modern browsers, but test it on your supported targets. Python's standard library has no grapheme segmenter. The third-party regex module supports \X for grapheme clusters, and some projects use the grapheme package. Pick a maintained library, and check that its Unicode data is recent enough for the emoji you expect.

Escapes: writing code points in source code

Source files are often limited to what you can type or what an encoding can carry. Escapes let you name any code point using plain ASCII. Each language chose its own syntax, which is a constant source of mistakes.

Context	Syntax	Example for é (U+00E9)	Example for 😀 (U+1F600)
JavaScript string	\uXXXX	é	😀
JavaScript (ES2015+)	\u{X...}	\u{e9}	\u{1F600}
JavaScript, Latin-1 only	\xXX	\xe9	not possible
Python 3 string	\uXXXX, \UXXXXXXXX	é	\U0001F600
Python by name	\N{NAME}	\N{LATIN SMALL LETTER E WITH ACUTE}	\N{GRINNING FACE}
HTML hex reference	&#xXXXX;	é	😀
HTML decimal reference	&#NNNN;	é	😀
CSS	\XXXX followed by a space or non-hex character	\E9	\1F600

I verified the JavaScript and Python rows by running them. In Node.js, "\u{1F600}" === "😀" and "😀" === "😀" were both true, and "\x41\xe9" produced Aé. In Python 3.13, "\N{GRINNING FACE}", "\U0001F600" and the literal emoji printed the same character.

JavaScript details

The \u{...} form takes one to six hex digits and accepts any code point up to U+10FFFF, so you never need to compute surrogates by hand. The older four-digit form only reaches the BMP, which forces you to write a surrogate pair for emoji. Both produce the same string. The \x form takes exactly two digits and is limited to U+0000 through U+00FF.

String.fromCharCode works on code units, so it accepts 0xD83D and 0xDE00 separately and joins them into a valid pair. String.fromCodePoint takes whole code points and throws a RangeError for values above 0x10FFFF.

Python details

Python has both a four-digit \u and an eight-digit \U, and you must supply all digits. ὠ0 is read as ὠ followed by a literal zero, which is a classic bug. The \N{...} form uses the official character name and is the most readable choice in source code, since a reader can understand \N{ZERO WIDTH SPACE} without a lookup table. Remember that these escapes apply in normal string literals. A raw string such as r"é" keeps the backslash and the letters.

HTML and CSS

Numeric character references work in HTML text and attribute values. Hex references need the x after the hash. They are handy for characters you cannot type and for keeping an ASCII-only source file, though in a UTF-8 document you can usually write the character directly. If you do, make sure the document declares its encoding with <meta charset="utf-8">, covered in the mojibake section.

CSS escapes appear mostly in the content property and in selectors. A CSS escape is a backslash followed by up to six hex digits. Because the digits run on, a space after the escape ends it, and that space is swallowed. Writing content: "\1F600 text" gives an emoji followed by the word. If the next character is itself a hex digit, the space or a six-digit padded form is mandatory, otherwise \E9a is read as the single code point U+0E9A. Our CSS formatting and architecture guide covers where content strings fit into maintainable stylesheets.

JSON and other formats

JSON strings use \uXXXX and have no \u{...} form, so a supplementary character is written as a surrogate pair. JSON.stringify("\uD83D") produced "\ud83d" in my test, an escaped lone surrogate. That output is legal JSON text but not a valid Unicode string, and other parsers may reject it. Python's json.dumps escapes all non-ASCII by default, writing the emoji as 😀. Pass ensure_ascii=False for readable output, as our JSON minification and payload size guide explains when it discusses size effects. Our JSON formatting guide covers the surrounding syntax rules.

Converting text to code points and back

The conversion has two directions. Text to code points means iterating by code point and printing each number. Code points to text means parsing numbers and building a string. The steps below are the same in every language, and the main trap is iterating by code unit instead.

JavaScript

const toCodePoints = (t) =>
  Array.from(t, (c) => "U+" + c.codePointAt(0).toString(16).toUpperCase().padStart(4, "0"));
const fromCodePoints = (list) =>
  list.map((u) => String.fromCodePoint(parseInt(u.slice(2), 16))).join("");
const toEscapes = (t) =>
  Array.from(t, (c) => {
    const n = c.codePointAt(0);
    return n < 0x80 ? c : n > 0xffff ? `\\u{${n.toString(16)}}` : `\\u${n.toString(16).padStart(4, "0")}`;
  }).join("");

const text = "Hi 世界 😀";
const cps = toCodePoints(text);
console.log(cps.join(" "));
console.log(fromCodePoints(cps) === text);
console.log(toEscapes(text));

Output from Node.js 22:

U+0048 U+0069 U+0020 U+4E16 U+754C U+0020 U+1F600
true
Hi 世界 \u{1f600}

Array.from(t, fn) iterates by code point, so the emoji appears once as U+1F600 and not as two surrogate halves. Always call codePointAt(0) on each element of that array, never charCodeAt.

Python

import unicodedata as ud

def to_cps(t):
    return [f"U+{ord(c):04X}" for c in t]

def from_cps(items):
    return "".join(chr(int(u[2:], 16)) for u in items)

def to_escapes(t):
    return "".join(
        c if ord(c) < 0x80 else (f"\\U{ord(c):08x}" if ord(c) > 0xFFFF else f"\\u{ord(c):04x}")
        for c in t
    )

t = "Hi 世界 😀"
c = to_cps(t)
print(" ".join(c))
print(from_cps(c) == t)
print(to_escapes(t))
print([ud.name(x) for x in "A😀é"])

Output from Python 3.13:

U+0048 U+0069 U+0020 U+4E16 U+754C U+0020 U+1F600
True
Hi 世界 \U0001f600
['LATIN CAPITAL LETTER A', 'GRINNING FACE', 'LATIN SMALL LETTER E WITH ACUTE']

Python's ord and chr are inverses on code points, and unicodedata.name gives the official name. The module reported Unicode data version 15.1.0 in this runtime, which tells you why a brand-new emoji may return "?" as a default name in an older interpreter. Your Unicode database is only as new as your runtime.

Sizes in each encoding

Python also shows the three encodings side by side. For "A😀é" the encoded lengths were:

Encoding	Bytes	Why
UTF-8	7	1 + 4 + 2
UTF-16 (little-endian, no BOM)	8	2 + 4 + 2
UTF-32 (little-endian, no BOM)	12	4 + 4 + 4

UTF-8 is the common choice on the web and in files because it is compact for ASCII-heavy text and has no byte-order problem. UTF-16 is how JavaScript, Java and Windows APIs hold strings in memory. UTF-32 is simple, with one unit per code point, but it wastes space.

Reading escapes back into text

Going from an escaped string to real text depends on the escape dialect. In JavaScript, a string literal is decoded by the parser, and JSON.parse decodes JSON escapes. Do not use eval to decode strings. In Python, "\\u4e16\\u754c".encode().decode("unicode_escape") returned 世界 in my run. That codec has a catch: it treats the input as Latin-1 text with escapes, so it garbles strings that already contain non-ASCII characters. For JSON data, use a JSON parser. For data you generated yourself, prefer a format with a real parser over hand-rolled escaping.

Real-world use cases

Debugging a mystery character. A string fails to match, a CSV import chokes, or a name looks right but compares unequal. Printing the code points exposes a non-breaking space (U+00A0) that looked like a space, or a Cyrillic letter in a Latin word.

Writing ASCII-only source files. Some build pipelines, legacy properties files and email-safe templates need escapes for anything outside ASCII. Converting text to escapes produces a portable source.

Internationalisation testing. Testers feed strings with accents, right-to-left text, emoji and long words into forms. Knowing the code points lets them write test cases that target one behaviour, such as supplementary-plane characters or combining marks.

Database and API limits. A field declared as 50 characters may count bytes, code units or code points depending on the system. Knowing which one is vital when users type emoji and non-Latin text. Databases that once supported only the BMP in their "utf8" setting are a famous trap: some systems offered a three-byte variant that rejects four-byte characters such as emoji, so check what your column type and connection charset actually accept.

Identifiers and usernames. Systems that allow Unicode usernames must decide on normalization and which categories are permitted. Doing it wrong allows lookalike accounts, covered in the security section.

Search and sorting. Search engines normalise and case-fold text before indexing. Application-level search that skips this step misses matches that users consider obvious.

Text processing pipelines. Cleaning scraped data, deduplicating records and detecting invisible characters in copied text all start with looking at code points. If you compare two exports of the same data, our guide to code diff and comparing files shows how a diff can highlight differences that are invisible to the eye.

Education and curiosity. Looking up the code point of a symbol, converting a word to its numbers, or finding the name of a character is an easy way to learn how text works.

Common mistakes and pitfalls

• Treating length as the number of characters. In JavaScript it counts UTF-16 units. In Python it counts code points. Neither counts graphemes.
• Splitting a string by index. slice, substring and split("") in JavaScript can cut a surrogate pair. Truncate by grapheme when the text is shown to people.
• Using charCodeAt on emoji. It returns the high surrogate only. Use codePointAt.
• Comparing strings without normalizing. Composed and decomposed forms look identical and compare unequal.
• Assuming one case mapping. Uppercasing "straße" gives "STRASSE", which is a different length. Some letters have context or language dependent mappings.
• Confusing encoding and character set. "Unicode file" is not a format. Say UTF-8, UTF-16 or UTF-32.
• Forgetting the encoding declaration. A browser or tool guessing the wrong encoding produces mojibake.
• Writing ὠ0 in Python. Use \U0001F600 or \N{GRINNING FACE}.
• Stripping all non-ASCII to "clean" input. This destroys names in most of the world's languages. Validate by category instead.
• Trusting visual identity. Two strings that look the same can be different code points. This is the basis of spoofing attacks.
• Ignoring invisible characters. Zero width spaces and byte order marks survive copy and paste and break comparisons, parsers and file names.
• Relying on a regular expression's dot. Without the u flag, . matches one code unit. In my test, "😀".match(/./)[0].length was 1, and with the flag it was 2 units, one whole code point.

Step-by-step: convert text to Unicode with DevUtilX

When you need an answer in seconds and do not want to write a script, use the DevUtilX Text to Unicode tool. It runs in your browser, so the text you paste is processed locally and nothing is uploaded.

1. Open the Text to Unicode tool.
2. Type or paste the text you want to inspect. A short sample is easiest when you are hunting for one odd character.
3. Run the conversion and read the output. Each character is shown as its Unicode value, so a hidden character that looked like a space reveals itself by its number.
4. Compare the values against the table of characters you expected. A Cyrillic "а" appears as U+0430 and not U+0061, even though the two look alike.
5. Copy the result into your code, test case or bug report. If you need escapes for a source file, use the escaped representation that matches the language you are writing.
6. To go the other way, paste escape sequences or code point values into the tool, or into a short script from the examples above, and confirm that the round trip returns the original text.

Treat the tool as a magnifying glass for text. When two strings behave differently, convert both and compare the numbers. The first difference you find is usually the cause. If the strings are long, a text diff on the converted output makes the difference stand out. Because the work happens locally, you can inspect internal text, such as a customer's name that failed to match, without sending it anywhere, though you should still avoid pasting secrets into any tool.

Case mapping, folding and collation

Text comparison involves three separate operations that people often blur together.

Case mapping is not one to one

English teaches that each lowercase letter has one uppercase partner. Unicode shows that this is not universal.

• German ß uppercases to SS. In Node.js, "straße".toUpperCase() returned "STRASSE", which is one character longer.
• Turkish and Azerbaijani distinguish a dotted and a dotless i. In my test, "I".toLocaleLowerCase("tr") returned "ı" (dotless), and "i".toLocaleUpperCase("tr") returned "İ" (dotted capital). The default "I".toLowerCase() returned "i". If you uppercase the word "title" with the wrong locale, you break it for Turkish users, and if you lowercase "FILE" with the Turkish locale, you get "fıle".
• The capital İ (U+0130) lowercases in the default locale to two code points, i plus a combining dot above (U+0307). JavaScript returned ["69","307"] and Python gave b'i\\u0307'.
• Some letters map to more than one letter in uppercase. Python's "ᾳ".upper() returned two characters, ΑΙ.

Case folding for comparisons

For case-insensitive comparison, lowercasing is not the right tool. Unicode defines case folding, which is designed for matching. Python has str.casefold(), and "ß".casefold() returned "ss", so "STRASSE".casefold() == "straße".casefold() was true. JavaScript has no casefold. A common approximation is to normalize and then compare with localeCompare using a sensitivity option, or Intl.Collator. Note that "Straße".toLowerCase() === "STRASSE".toLowerCase() was false in my run, which proves the point.

Collation: sorting depends on the language

Sorting by code point is fast and predictable, and wrong for people. In my JavaScript test, the default ["z","ä","a"].sort() returned ["a","z","ä"], because ä is U+00E4, a higher number than z. With the German locale, the result was ["a","ä","z"]. Sorting is also language specific. new Intl.Collator("sv").compare("z","ä") returned -1, meaning Swedish puts ä after z, while the German collator returned 1.

Use Intl.Collator or localeCompare with an explicit locale for any list shown to humans. Use code point order only for internal keys where stability matters more than linguistic correctness.

Invisible and misleading characters

Some characters have no glyph at all. They are legitimate and useful, and they cause grief when they appear by accident.

Character	Code point	Category	Role
Zero width space	U+200B	Cf	Allows a line break with no visible gap
Zero width non-joiner	U+200C	Cf	Prevents joining in scripts that join letters
Zero width joiner	U+200D	Cf	Requests joining, used by emoji sequences
Word joiner	U+2060	Cf	Prevents a break
Byte order mark / zero width no-break space	U+FEFF	Cf	Marks byte order at the start of a file
Non-breaking space	U+00A0	Zs	Space that does not wrap

Two facts from my runs matter in practice. First, "​hi".trim().length was 3: trim did not remove the zero width space, because JavaScript treats it as a format character and not as whitespace. Second, "hi".trim().length was 2, since the BOM, also known as the zero width no-break space, is counted as whitespace by trim. The behaviours are not intuitive, and you cannot guess them from how the characters look.

The byte order mark

A BOM at the start of a UTF-8 file is the three bytes EF BB BF. UTF-8 has no byte order issue, so a BOM is unnecessary, but some Windows tools add one. Programs that do not expect it treat it as data. A CSV header may become id instead of id, a shell script may fail on its first line, and a JSON parser may reject the file. RFC 8259 tells producers not to add a BOM to JSON text, and parsers are allowed to reject one.

Decoders vary in what they do. In Node.js, new TextDecoder("utf-8") removed the BOM from the three bytes plus "hi" and returned a string of length 2, while new TextDecoder("utf-8", { ignoreBOM: true }) kept it and returned length 3. In Python, b"\xef\xbb\xbfcafé" decoded with "utf-8" retains the invisible character, while the "utf-8-sig" codec removes it. When you read files from unknown sources, use utf-8-sig in Python.

Cleaning invisible characters

This JavaScript helper reports and removes the usual suspects. Do not apply the removal blindly: a ZWJ inside an emoji sequence or a ZWNJ in Persian text is meaningful, and deleting it changes the text.

const hasInvisible = (s) => /[\p{Cf}\p{Cc}]/u.test(s.replace(/[\n\t]/g, ""));
console.log(hasInvisible("a​b"), hasInvisible("ab"));  // true false

const clean = (s) => s.normalize("NFC").replace(/[​-‍⁠]/g, "");
console.log(JSON.stringify(clean("ca​fé")));  // "café"

The Python equivalent lists the characters by name, which makes bug reports clear:

import unicodedata as ud

def inv(s):
    return [f"U+{ord(ch):04X} {ud.name(ch, '?')}" for ch in s
            if ud.category(ch) in ("Cf", "Cc") and ch not in "\n\t"]

print(inv("a​b"))
# ['U+200B ZERO WIDTH SPACE', 'U+FEFF ZERO WIDTH NO-BREAK SPACE']

For file names, search keys and identifiers, strip invisible characters and normalize. For free text such as messages and documents, keep them, because emoji and several scripts depend on joiners.

Security: lookalikes and bidirectional text

Unicode's breadth makes it possible to write text that reads one way and means another. This section stays conceptual. The goal is to know what to defend against, not how to craft an attack.

Homoglyphs and confusables

Different scripts contain letters that look the same or nearly so. The Cyrillic small letter a, U+0430, looks like the Latin small letter a, U+0061. In Node.js, "a" === "а" was false, and Python's unicodedata.name returned CYRILLIC SMALL LETTER A versus LATIN SMALL LETTER A. A human cannot tell the strings apart on screen. This enables lookalike domain names, impersonated usernames, and fake file names.

The Unicode Consortium publishes a confusables data set and a technical report on security mechanisms that implementations can use. Practical defences include:

• Restrict identifiers to one script, or to approved combinations of scripts.
• Normalize and compare usernames in a canonical form, so that two accounts cannot differ only by composed versus decomposed text.
• For domains, rely on the browser's handling of internationalised domain names, which can show the encoded form when a name mixes scripts in suspicious ways.
• Reserve sensitive names such as "admin" in a skeleton form, so lookalikes are rejected too.
• Show users the actual code points or a warning when high-risk text, such as a payee name, contains mixed scripts.

Bidirectional text controls

Scripts such as Arabic and Hebrew are written right to left, and the Unicode bidirectional algorithm decides how mixed-direction text is arranged on screen. It also provides explicit control characters, such as the right-to-left override (U+202E), that force the order of what follows. Because these characters are invisible, text containing them can display in a different order than it is stored.

The risk is that what a reviewer sees is not what a compiler or file system processes. A file name can be made to display with its extension in a misleading position. Source code containing bidirectional controls inside comments or strings can look as though it does something other than what it does, a class of issue often discussed under the name "Trojan Source". In my test, the string "abc‮def" had length 7 in JavaScript, which is one more than its visible letters, and the control is invisible.

Defences are simple in principle:

• Reject or strip bidirectional control characters (U+202A to U+202E and U+2066 to U+2069) from identifiers, file names and other fields where they have no purpose.
• Configure editors, code review tools and CI to flag them in source. Many editors can show invisible characters.
• Allow them only where legitimate bidirectional text requires them, and then prefer the isolate controls designed for safer embedding.

const bidi = /[‪-‮⁦-⁩]/;
console.log(bidi.test("abc‮def"), bidi.test("abc"));  // true false

Other security notes

• Normalize before you validate. If validation runs before normalization, an attacker can pass a form that is rejected in one shape and accepted in another. Normalize first, then check, then store the normalized value.
• Length limits need a defined unit. A limit of 255 bytes is not 255 characters. Measure in the unit that your storage and protocol use.
• Beware lossy compatibility forms. NFKC can turn a fullwidth or superscript character into an ASCII lookalike, which is useful for matching and risky for display.
• Escape on output. Unicode does not remove the need for output escaping in HTML, SQL and shell contexts. Characters outside ASCII can still be dangerous when mishandled by a naive filter.
• Sanitise logs. Control characters in user input can mislead people reading log files. Escape them when writing logs.

Performance, memory and accessibility

Performance and memory

• Strings cost by encoding. UTF-8 text that is mostly ASCII is compact. Text in scripts that use three-byte sequences costs more in UTF-8 than in UTF-16. Measured on "é😀", UTF-8 used six bytes, with encodeURIComponent producing %C3%A9%F0%9F%98%80. Do not choose an encoding based on one sample. Measure your own content.
• Iteration by code point allocates. [...text] creates an array, which is fine for short strings and expensive for megabytes. Use the iterator directly in a for...of loop to avoid the copy.
• Segmenters are slower than indexing. Grapheme segmentation does real work. Run it where you need it, such as when a user edits a field, and not in a loop across a million records.
• Normalization has a fast path. Strings that are already normalized are cheap to check in many implementations, but do not assume that. Normalize at the boundary where text enters your system and store the result.
• Precompute keys. For sorting or searching large lists, compute a normalized, folded key once instead of inside the comparator.

Accessibility

Screen readers read what the text says, and Unicode choices change what they say.

• Do not use decorative Unicode as styled text. Mathematical alphanumeric letters (the so-called bold or script fonts) are different code points, and many screen readers read them as symbols or letter by letter, so a styled headline may be unreadable. Use CSS for styling.
• Emoji are announced by name. A string of ten emoji is read as ten descriptions. Use them sparingly, and put meaning in words.
• Emoticons and ASCII art are read as punctuation. Provide a text alternative where they carry information.
• Set the language. The lang attribute lets assistive technology and the browser pick correct pronunciation, hyphenation and case rules, including the Turkish i behaviour above. Our HTML formatting and semantic markup guide covers attributes such as lang and dir.
• Declare direction. Use dir for right-to-left content instead of inserting invisible control characters.
• Keep copy-paste honest. Invisible characters that disrupt screen readers, such as zero width spaces inserted to break words, hurt navigation by word. Avoid them in content.

Mojibake and encoding problems

Mojibake is text decoded with the wrong encoding. The bytes are fine, the interpretation is wrong. It is the most common Unicode problem you will meet, and it has recognisable patterns.

The classic case is UTF-8 bytes read as a single-byte encoding. The word "café" is the bytes 63 61 66 C3 A9 in UTF-8. A decoder using windows-1252 maps C3 to "Ã" and A9 to "©", so you see "café". In my Node.js run, decoding the UTF-8 bytes of "café" as windows-1252 gave exactly that, and Python's decode("latin-1") gave the same result.

Because the damage is mechanical, it can often be reversed if no byte was lost. Python round-trips it:

raw = "café".encode("utf-8")
broken = raw.decode("latin-1")
print(broken)                                    # café
print(broken.encode("latin-1").decode("utf-8"))  # café

Reversal works only if the text went through a lossless single-byte decoding. If the broken text was saved, re-encoded, or had characters replaced with question marks, the original is gone.

Another common pattern is the replacement character, a diamond with a question mark, U+FFFD. Decoders emit it when they meet bytes that are invalid in the declared encoding. In Python, b"\xe9".decode("utf-8") raised an error ("unexpected end of data"), and with errors="replace" it returned the replacement character instead. That byte is the Latin-1 é, so the data was probably not UTF-8 in the first place.

The reverse failure also exists. Python's "é".encode("ascii") raised a UnicodeEncodeError saying that the ordinal is not in range 128. That error is a feature, since it tells you the target cannot hold your data.

Where encodings get declared

An encoding must be agreed at every hop, and any hop that guesses can break the chain.

• Files: an editor setting, sometimes a BOM. Save source files as UTF-8.
• HTTP: the Content-Type header can carry charset=utf-8. For HTML, also include <meta charset="utf-8"> near the top of <head>.
• Databases: the database, table, column and connection each have a character set. A mismatch at any one of them produces mojibake or lost characters.
• Command line: terminal and locale settings decide how bytes are displayed.
• Email, CSV and legacy exports: often use older encodings. CSV opened in a spreadsheet application may be interpreted with a regional default encoding unless it has a BOM or the import dialog is told otherwise.
• URLs: non-ASCII characters are percent-encoded as UTF-8 bytes. encodeURIComponent("é😀") returned %C3%A9%F0%9F%98%80.

Troubleshooting guide

Work from the symptom to the cause.

Symptom: letters like "é" or "’" appear in place of accents and quotes

Cause. UTF-8 bytes were decoded as a single-byte encoding such as Latin-1 or windows-1252.
Fix. Find the layer that guessed. Declare UTF-8 in the file, header, meta tag or connection. To repair data, reverse the wrong decoding as shown above, and check the result on a sample first.

Symptom: black diamonds with question marks, or ? characters

Cause. The decoder met invalid bytes and substituted U+FFFD, or an encoder could not represent a character and wrote a question mark.
Fix. Check the real encoding of the source. Question marks written during encoding are permanent loss, so fix the pipeline and re-import the original.

Symptom: two identical-looking strings are not equal

Cause. Composed versus decomposed forms, a lookalike character from another script, or an invisible character.
Fix. Print the code points of both strings, as in the examples above, and find the first difference. Normalize to NFC at input boundaries. If the cause is an invisible character, remove it where it has no purpose.

Symptom: length says 2 for one emoji, or a character counter is off

Cause. The language counts code units or code points, not grapheme clusters.
Fix. Use Intl.Segmenter in JavaScript or a grapheme library in Python. Decide which unit the business rule means, and document it.

Symptom: a string has a lone box character after slicing or reversing

Cause. A surrogate pair was split.
Fix. Iterate by code point or grapheme. Use toWellFormed() where available to repair, and fix the code that split the string.

Symptom: the first column name has an invisible prefix

Cause. A UTF-8 byte order mark at the start of the file.
Fix. Read with utf-8-sig in Python or strip U+FEFF at the start. Ask the producer to export without a BOM if possible.

Symptom: trim() does not remove the odd space

Cause. The character is not whitespace to the language. Zero width space is a format character, and non-breaking space handling varies by function.
Fix. Identify the code point first, then remove it explicitly.

Symptom: sorting puts accented letters at the end of the list

Cause. Sorting by code point.
Fix. Use Intl.Collator or localeCompare with the user's locale.

Symptom: Turkish users report broken uppercase or lowercase text

Cause. The default case mapping was applied to text in a locale with different rules.
Fix. Pass the locale to toLocaleUpperCase and toLocaleLowerCase, and use case folding for identifiers and comparisons.

Symptom: a database rejects an emoji

Cause. The column or connection uses a character set that cannot store four-byte UTF-8 sequences.
Fix. Switch to a full UTF-8 character set at the column, table and connection level, and test with a supplementary-plane character.

Symptom: \u{...} is a syntax error

Cause. The runtime or tooling predates the ES2015 syntax, or you are writing JSON, which has no such form.
Fix. Write a surrogate pair in JSON and in old toolchains, or upgrade the tool that parses the file.

Best practices and checklist

• Use UTF-8 everywhere you control: files, databases, APIs and terminals. Declare it explicitly.
• Decode at the edges. Convert bytes to strings when data enters your program, and back when it leaves. Inside, work with strings only.
• Normalize to NFC at input boundaries. Store the normalized value, and compare normalized values.
• Choose the unit deliberately. Bytes for storage and protocols, code points for algorithms, graphemes for anything a person sees.
• Use codePointAt and fromCodePoint, never the code-unit methods, for anything that may include emoji.
• Match with \p{...} property escapes and the u flag instead of hand-written ranges.
• Compare with case folding and collation, not with toLowerCase or code point order.
• Control invisible characters. Strip or reject them in identifiers and file names. Keep them where scripts and emoji need them.
• Reject bidirectional controls in identifiers, names and source code unless needed.
• Restrict scripts in identifiers and compare in a confusable-aware form for high-risk names.
• Keep escapes ASCII-safe and language-correct. Use \u{...} in modern JavaScript, \N{...} in Python, and numeric references in HTML.
• Test with hard strings. Include an accented letter in both forms, a flag, a ZWJ emoji, a right-to-left word, a combining mark, a lone surrogate and a zero width space.
• Log code points when debugging instead of guessing from the display.
• Record your runtime's Unicode version when behaviour depends on new characters.

A short pre-release checklist:

1. Is every file, response header and database connection declared as UTF-8?
2. Is text normalized at the boundary?
3. Do length limits state their unit and match what the storage layer counts?
4. Are identifiers checked for scripts, invisible characters and bidirectional controls?
5. Do sorting and case-insensitive search use locale-aware tools?
6. Do the tests include an emoji with a modifier, a flag, and a combining accent?
7. Does a character counter or truncation function use grapheme segmentation?

Comparison: tools and approaches for working with Unicode text

Approach	Best for	Strengths	Limits
DevUtilX Text to Unicode	Quick inspection and conversion of a snippet	No setup, runs locally, shows hidden characters	Manual, one snippet at a time
JavaScript codePointAt and Array.from	Browser and Node.js code	Built in, no dependencies	Code points only, graphemes need Intl.Segmenter
Python ord, chr, unicodedata	Scripts and data pipelines	Names, categories and normalization in the standard library	No built-in grapheme segmentation
Intl.Segmenter and Intl.Collator	User-facing counting, truncation and sorting	Locale-aware	Runtime support must be checked
Command-line tools such as xxd or iconv	Bytes in files and re-encoding	Show raw bytes, convert encodings	Do not explain characters
Editor features	Spotting invisible characters in source	Visible while you edit	Settings differ by editor
Third-party libraries	Confusable detection, grapheme segmentation outside JavaScript	Depth and specialised data	Another dependency to keep updated

No single approach covers everything. A typical debugging session uses the browser tool to spot the odd value, a script to scan thousands of records for the same pattern, and a locale-aware API in the fix.

FAQ

What is a Unicode code point?
It is a number assigned to a character, written in hex with the prefix U+. For example, U+0041 is the capital letter A and U+1F600 is the grinning face emoji. The valid range runs from U+0000 to U+10FFFF.

Is Unicode the same as UTF-8?
No. Unicode assigns numbers to characters. UTF-8 is one way to store those numbers as bytes. UTF-16 and UTF-32 are others.

Why is the length of an emoji string greater than one in JavaScript?
JavaScript counts UTF-16 code units. Characters outside the Basic Multilingual Plane use two units, and sequences such as the family emoji use many more. The 😀 emoji had length 2, and the family emoji had length 11 in my test.

What is the difference between a code point and a grapheme cluster?
A code point is one Unicode number. A grapheme cluster is what a reader sees as a single character, which can be made of several code points such as a letter plus an accent, a flag, or a ZWJ emoji sequence.

How do I get the code point of a character in JavaScript and Python?
In JavaScript, use "😀".codePointAt(0), which returns 128512. In Python, use ord("😀"). To go back, use String.fromCodePoint(128512) or chr(128512).

What is a surrogate pair?
It is two UTF-16 code units, a high surrogate followed by a low surrogate, that together represent one code point above U+FFFF.

Why do "é" and "é" compare as different strings?
One is a single composed code point, and the other is an e followed by a combining accent. Normalize both to NFC before comparing.

Which normalization form should I use?
Use NFC for storage and comparison in most applications. Use NFKC for search keys and identifiers when you want compatibility variants to match, but not for content you must preserve exactly.

What does \u{1F600} mean and where does it work?
It is the ES2015 JavaScript escape for a code point with any number of hex digits. It works in modern JavaScript strings, templates and regular expressions with the u flag, but not in JSON.

Why does "café" appear instead of "café"?
The UTF-8 bytes were decoded with a single-byte encoding such as windows-1252 or Latin-1. Declare UTF-8 at every layer, and reverse the wrong decoding if the data was not otherwise changed.

What is a zero width space and why is it a problem?
U+200B is an invisible character that permits a line break. It survives copying, breaks equality checks and is not removed by JavaScript's trim. Detect it by looking at code points.

Can lookalike characters be dangerous?
Yes. A Cyrillic letter can look identical to a Latin one, which enables impersonation in usernames, domain names and file names. Restrict scripts, normalize and use confusable checks for high-risk identifiers.

Is it safe to paste text into an online converter?
Only if the tool works locally, as DevUtilX does. Even so, avoid pasting passwords, tokens or personal data into any web page.

Why does uppercasing change the length of a string?
Some characters expand. German ß becomes SS, as "straße".toUpperCase() showed in my run. Length-dependent code must not assume case changes preserve length.

Conclusion

Unicode makes text a layered thing: code points name characters, encodings store them as bytes, and graphemes are what people actually see. Most bugs come from mixing those layers, and most fixes come from picking the right layer on purpose. Normalize at the edge, count in the unit that matters, compare with locale-aware tools, treat invisible and lookalike characters as input to validate, and declare UTF-8 everywhere. When you want to see exactly what is inside a string, paste it into the Text to Unicode tool and read the numbers. To go further, read our guides on JSON formatting and HTML formatting and semantic markup, where encoding declarations and escapes appear in everyday work.

Further reading

• Unicode – Wikipedia
• The Unicode Consortium
• String – MDN Web Docs
• UTF-8 – Wikipedia
• UTF-16 – Wikipedia
• Unicode equivalence – Wikipedia
• Mojibake – Wikipedia