Characters seem like an easy concept to grasp at first glance. , "Java"has a
primitive type char and an object wrapper Character, and these are pretty
close to what we mean when we talk about characters. However, these data types
aren't fully inclusive of the set of characters we might actually encounter in
programs. There are good historical / compatibility reasons for this
inconsistency, detailed in the Character
javadocs,
but I'd like to share a debugging story to highlight the difference between the
Java types and what we usually mean when we say "character".
Recently I received a bug report for
REPLy, where the character represented by
"\ud800\udf30" [1], when pasted as
input, would immediately exit the program (a nasty bug, to be sure). After a
bit of yak shaving, Li-Hsuan and I traced the
problem down to a class in Jline responsible
for converting bytes from an input stream into characters. Java has a class
java.io.InputStreamReader that's responsible for doing exactly this, but it
wasn't being used, possibly for historical reasons. The class had been taken
from the Apache Harmony codebase, and it
correctly handled many non-ASCII UTF-8 characters. All of the characters that
worked, however, were a single char in length. The broken ones were non-BMP
characters: this was the crux of the problem.
BMP here isn't the image file
format, as I learned, but rather
the Basic Multilingual
Plane.
The BMP characters are up to 2 bytes in length (values 0x0000-0xffff), so
there's a good bit of coverage there. Characters from the Chinese, Thai, even
Mongolian alphabets are there, so if you're not an encoding expert, you might
be forgiven if your code only handles BMP characters. But all the same,
characters like the one in
question won't
be correctly handled by code that assumes it'll fit into two bytes.
Back to Java: a char is defined to hold 16 bits of
data,
and the Character type is just a wrapper around that. And as we've just
learned, that's precisely enough space to hold the full range of BMP
characters. But when it comes to non-BMP characters, which have values greater
than 0xffff, we're out of luck.
The right approach is to bridge the language-level chars to semantic characters
by using an API like Character.codePointAt(char[], int)
to get one real character starting at a given index in the input array.
The output is an int, which is what we need to have enough room for these
non-BMP characters. Impressively, the class with the bug was already using an array
of chars; it just turned out to only be using one element (of a 1-char array) instead
of allowing for multi-char characters. This ended up causing an error case
where the alternate InputStreamReader would return -1 to indicate the end of
the stream. The actual fix was easy,
but as usual, understanding the context surrounding the issue was less so.
As I write this, there are currently 417,000 Google results for non-bmp
character bug. This kind of confusion is definitely not limited to Java or
even the JVM. If you're interested in learning more about how Unicode works,
the wikipedia article on Unicode character
planes is as
good a place to start as any. There's much more complexity here than I've
described in this short post, but if this prevents at least one future mistaken
assumption about the size of a character, I'm happy.
[1] I'll settle for linking the description here since something in my toolchain, likely either my browser or my font set, doesn't support it within this blog post.
