The windows operating system has held out for the longest time from using the UTF-8 encoding as the default standard for all document files. Thus you often had different code pages or encodings for different languages and sometimes even for different variants of the same language. Because different countries need different symbols for their currency.

Case in point: More often than not the windows in the UK used a different encoding than the windows in the USA. Because the UK users needed the £ (Pound Sterling) symbol for their currency.

Why is this happening? Glad you asked.

Back in the 1950s to 1970s when the electronic computers were developed the first punch cards used a seven binary bit encoding for control, letters, digits and punctuation. Using binary math that gives a range from 0 to 127. And these 128 character codes are known as the ASCII character set.

But pretty soon the computers got “standardized” on 8-bit bytes. Which can store up 256 discreet values. So there were another 128 code points available for more characters. And this is where it gets messy and confusing, because each country defined their own encodings in the 128 to 255 range. So one country would define the code point 192 to be Ä, while another would have Æ, a third would have £ and other would leave it undefined.

When the 1990s and the Internet rolled around, the exchange of digital computer documents across country and language borders started to become more common. And all of a sudden recipients were seeing weird garbage instead of clean text. (Who can still remember the “good old days” of e-mail exchange when foreign correspondents used their accustomed accents and make the message hard to read for you?) Not to mention the languages with non-roman scripts. That was when Unicode was initially conceived. They started out with 2-byte (16 bit) characters, but quickly realized that that was still not enough code points. Especially with the Japanese, Korean and Chinese characters. So they expanded to 4-byte (32 bit) characters.

But the developers soon realized that about 70% of all text data was wasting a lot of precious space. Remember, at that time a 250MB hard disk was massive in size and hugely expensive! So by the late 1990s the Unicode Transformation Format – 8-bit or UTF-8 encoding was established as the default standard on the Internet. However Microsoft did not support UTF-8 in their internal interface and documentation until May 2019.

The benefit of UTF-8 is that it encodes each character in either 1, 2, 3 or 4 bytes. So if your text is mostly in English or west European languages you are not wasting a lot of storage space. But that is also the reason why you sometimes get those weird multiple characters in place of accented letters, special symbols or even emojis. Because the author saved the document in their local encoding instead of in UTF-8.

Especially on older systems or with older fonts you might see a white rectangle with a question mark to indicate that the system is unable to render that specific character. Meanwhile the black diamond with a question mark (�) indicates that there is a problem with the encoding and the “character” has an invalid code point.

Sorry for geeking out on you. But I had just recently performed a deep dive into the whole character encoding mess history, and just wanted to share it with somebody.