Telecoms regulator Ofcom tells BT Group to cut the wholesale prices it charges for use of its lines

Developments in eye-tracking technology are helping advertisers predict what products will appeal to customers.

Facebook and Google tell a court in India they have complied with an order to remove "objectionable" material, amid ongoing legal moves on censorship.

Pop singer Cher Lloyd tells Panorama's Declan Lawn about being cyberbullied and her fears for her family's safety.

Olympus shareholders will get the chance to question management about the accounting scandal as the firm calls an emergency meeting.

MPs call on internet service providers to make greater efforts to remove violent extremist material.

The FBI investigates how activists linked to Anonymous obtained a recording of a phone call between US and UK police on their operations against hacking.

Computers store every piece of text using a “character encoding,” which gives a number to each character. For example, the byte 61 stands for ‘a’ and 62 stands for ‘b’ in the ASCII encoding, which was launched in 1963. Before the web, computer systems were siloed, and there were hundreds of different encodings. Depending on the encoding, C1 could mean any of ¡, Ё, Ą, Ħ, ‘, ”, or parts of thousands of characters, from æ to 品. If you brought a file from one computer to another, it could come out as gobbledygook.

Unicode was invented to solve that problem: to encode all human languages, from Chinese (中文) to Russian (русский) to Arabic (العربية), and even emoji symbols like or
; it encodes nearly 75,000 Chinese ideographs alone. In the ASCII encoding, there wasn’t even enough room for all the English punctuation (like curly quotes), while Unicode has room for over a million characters. Unicode was first published in 1991, coincidentally the year the World Wide Web debuted—little did anyone realize at the time they would be so important for each other. Today, people can easily share documents on the web, no matter what their language.

Every January, we look at the percentage of the webpages in our index that are in different encodings. Here’s what our data looks like with the latest figures*:


As you can see, Unicode has experienced an 800 percent increase in “market share” since 2006. Note that we separate out ASCII (~16 percent) since it is a subset of most other encodings. When you include ASCII, nearly 80 percent of web documents are in Unicode (UTF-8). The more documents that are in Unicode, the less likely you will see mangled characters (what Japanese call mojibake) when you’re surfing the web.

We’ve long used Unicode as the internal format for all the text Google searches and process: any other encoding is first converted to Unicode. Version 6.1 just released with over 110,000 characters; soon we’ll be updating to that version and to Unicode’s locale data from CLDR 21 (both via ICU). The continued rise in use of Unicode makes it even easier to do the processing for the many languages that we cover. Without it, our unified index it would be nearly impossible—it’d be a bit like not being able to convert between the hundreds of currencies in the world; commerce would be, well, difficult. Thanks to Unicode, Google is able to help people find information in almost any language.

Posted by Mark Davis, International Software Architect

Apple is granted a suspension of a sales ban imposed on some of its iPads and iPhones in Germany.

An EU data protection group writes to Google to ask them to suspend changes to their privacy policy.

Is it a bird? Is it a plane? No, it's a remote-controlled man-shaped plane. This and more in this week's tech news.