DevToolkit Pro

Scraped data shows up as ???. Chinese stored in MySQL looks mangled. PHP strlen("你好") returns 6 instead of 2. These problems almost never trace back to the database or browser. They trace back to not understanding what ASCII, Unicode, and UTF-8 actually are, and how they relate.

This guide breaks the three apart so the next time you see garbled text, you can tell whether it's a character set problem or an encoding problem in seconds. To inspect or convert characters on the spot, our Unicode encoder/decoder shows the code point and UTF-8 bytes for any text.

ASCII: The 7-Bit Era Of 128 Characters

ASCII (American Standard Code for Information Interchange) dates from the 1960s. It uses 7 bits per character for 128 total code points:

| Range | Content | Count | |-------|---------|-------| | 0-31 | Control characters (newline, tab, etc.) | 32 | | 32-126 | Printable characters (letters, digits, symbols) | 95 | | 127 | DEL | 1 |

'A' = 65 = 0x41
'a' = 97 = 0x61
'0' = 48 = 0x30
' ' = 32 = 0x20

ASCII solved English storage. But only English. German ä, Chinese 中, Japanese あ, Emoji, none of those fit.

Unicode: A Character Set For Every Language

Unicode is a giant table that assigns a unique number (code point) to almost every character in every language. The range runs from U+0000 to U+10FFFF, theoretically holding more than 1.1 million code points. Over 140,000 are assigned today.

U+0041  A           (Latin Capital Letter A)
U+4E2D  中          (CJK Unified Ideograph)
U+1F600 😀          (Grinning Face)
U+1F4A9 💩          (Pile of Poo)

Unicode specifies only "which character gets which number." It does not specify how that number is stored in memory. The byte-level representation is called an encoding, and UTF-8 is the most popular one.

UTF-8: Unicode's Variable-Length Encoding

UTF-8 is one implementation of Unicode. It uses 1 to 4 bytes per code point:

| Bytes | Code point range | Typical chars | Example | |-------|------------------|---------------|---------| | 1 | U+0000 - U+007F | ASCII | A → 0x41 | | 2 | U+0080 - U+07FF | Latin extended, Greek, Cyrillic | é → 0xC3 0xA9 | | 3 | U+0800 - U+FFFF | CJK characters | 中 → 0xE4 0xB8 0xAD | | 4 | U+10000+ | Emoji, ancient scripts | 😀 → 0xF0 0x9F 0x98 0x80 |

Variable length is the design win. English text stays as compact as ASCII. CJK characters cost an extra byte but actually exist. No waste.

UTF-8 Is Backwards Compatible With ASCII

This is UTF-8's smartest design choice. The first 128 code points (U+0000 to U+007F) each encode to one byte in UTF-8, and that byte's value is identical to ASCII:

ASCII 'A' = 0x41
UTF-8 'A' = 0x41   # same byte

Any pure ASCII text is automatically valid UTF-8. That's why HTML, email, and HTTP all default to UTF-8 today. Legacy English systems keep running without a single code change.

Character vs Byte: The "你好" Example

"你好" is 2 characters but takes 6 bytes in UTF-8:

Char:   你                好
Code:   U+4F60            U+597D
UTF-8:  E4 BD A0          E5 A5 BD
Bytes:  ----- 3 -----     ----- 3 -----

$ echo -n "你好" | xxd
00000000: e4bd a0e5 a5bd                                ......
$ echo -n "你好" | wc -c
       6

PHP and C strlen count bytes, so strlen("你好") returns 6. You need mb_strlen to get the character count of 2:

<?php
$s = "你好";
echo strlen($s);     // 6 (bytes)
echo mb_strlen($s);  // 2 (characters)

This distinction is the source of 90% of character-related bugs. JavaScript String.length counts UTF-16 code units, so a 4-byte Emoji like 😀 returns 2. That's another layer of the same trap.

Hex Dump Comparison

"A"   hex: 41                        # 1 byte
"中"  hex: E4 B8 AD                   # 3 bytes
"😀"  hex: F0 9F 98 80                # 4 bytes
"A中" hex: 41 E4 B8 AD                # mixed, 4 bytes total

UTF-8 decoders use the high bits of the leading byte to figure out the length:

• Leading byte 0xxxxxxx: 1-byte character
• Leading byte 110xxxxx: 2-byte character, followed by 1 10xxxxxx
• Leading byte 1110xxxx: 3-byte character, followed by 2 10xxxxxx
• Leading byte 11110xxx: 4-byte character, followed by 3 10xxxxxx

That's why UTF-8 is self-synchronizing. Start anywhere in a byte stream and you can locate the next character boundary by scanning at most a few bytes.

Common Encoding Bugs

1. Slicing in the middle of a multibyte character

// Wrong: substr on byte length can split a 3-byte sequence
const s = "你好世界";
s.substr(0, 3);  // may yield half of 你

Modern JavaScript slice counts by character and is safe. But raw Buffer operations, fetch streaming, and byte-level truncation still bite.

2. MySQL utf8 vs utf8mb4

MySQL's utf8 charset only holds 3-byte characters, so Emoji inserts fail silently or with an error. You need utf8mb4. MySQL 8.0 made utf8mb4 the default, but old databases are everywhere.

3. BOM showing up as content

A BOM (U+FEFF) at the file start tells the reader the file is UTF-8. But that BOM also leaks into output before PHP's header() call, triggering "headers already sent." Most of the time, save as UTF-8 without BOM.

When To Think About Encoding

• Before storing in a database: verify the table charset is utf8mb4 and the connection is utf8mb4 too
• Writing HTTP responses: Content-Type: application/json; charset=utf-8
• Parsing binary protocols: pin down byte order and encoding for each field
• Passing strings between languages: always UTF-8, never a locale-specific encoding

Do It In Your Browser

Debugging encoding issues needs hex dumps, code point lookups, and byte counts. You don't want to paste that data, especially the parts with credentials or user content, into a random website.

The Unicode Encoder/Decoder lists code points and byte sequences for every character. The Base64 Decoder confirms that Chinese embedded in Base64 survives the round trip. The Character Frequency tool helps verify a long text contains the expected characters. All three run entirely in your browser. Nothing leaves your machine.

The next time text looks wrong, read the hex first, then the code points, then the charset. The fix is usually simpler than it looks.