Character set and Java

by Hideyuki Inada, Capitola Computing Inc.

Introduction

Character set conversion is one of trickiest areas in Java. In this section, we will look at several areas that can cause character corruption problems if you are not aware of the underlying character set or encoding issues.

 

Character set used in Java and javac

First, let's take a look at some simple code that contains a Japanese string, and go through the compilation process to see what character sets are used in each of the file formats that are used in the typical development cycle.

In this example, the string is output to the console, and the codepage for the console is set to codepage 932 which is used for displaying the source code and the output. Codepage 932 is commonly known as Shift JIS and is the standard character set for personal computers used in Japan:

 

 

Shown below is the source code displayed to the same console using the "type" command:

 

intl1.java

 

If you compile intl1.java using javac, and run it with the "java" command, you'll see the following output on the console:

Output of intl1.class

 

As you can see:

l         Source code is encoded in codepage 932.

l         Output is generated in codepage 932.

 

At this point, we still do not know how the string is encoded in the class file.  A natural guess would be that those letters are encoded in codepage 932 as well.

If the strings are encoded in codepage 932, the string "ソフトウェアの国際化のデモ" should be stored as (shown in hexadecimal), and we should be able to find these in the class file (unless the literal strings are compressed):

835c 8374 8367 8345 8346 8341 82cc 8d91 8ddb 89bb 82cc 8366 8382

 

Let's dump the content to see if this assumption is true:

 

 

Unfortunately, the hexadecimal sequence that we are looking for is not in the class file. To make the investigation easier, let's change the source code to the following:

 

 

Hexadecimal representations of each letter in codepage 932 are shown in the table below:

 

Letter	Codepage 932	Unicode (UCS-2)	UTF-8
A	0x41	0x0041	0x41
B	0x42	0x0042	0x42
C	0x43	0x0043	0x43
ソ	0x835C	0x30BD	0xE3 0x82 0xBD
フ	0x8374	0x30D5	0xE3 0x83 0x95
ト	0x8367	0x30C8	0xE3 0x83 0x88
1	0x31	0x0031	0x31
2	0x32	0x0032	0x32
3	0x33	0x0033	0x33

Table 1

 

This way, the Japanese string is sandwiched between "ABC" and "123" and we should be able to figure out the encoding more easily.

 

Below is the result obtained by using tdump against the class file:

 

In the section following the address 0x90, we can see "ABC" and "123":

 

Looking at this result and the table 1, we can see that the string is stored in UTF-8 in the class file.   If you are interested in knowing the format used in the class file format, please refer to the Chapter 12 of "Java Virtual Machine" by Jon Meyer & Troy Downing.

 

Next, let's take a look at how these strings are processed inside the Java program.

We will do this by making changes intl2.java so that it will dump the content of the String object.

 

intl3.java

 

Output is shown below:

 

 

As shown above, during the execution of a Java program, each letter is treated as UCS-2.  In other words, each letter is treated as one "char" regardless of its character type. This is a big improvement over previous languages since you do not have to worry about checking the lead byte to determine the character set to branch out your code based on the character set.

To summarize, the following table shows how the code is encoded in each area of Java development.

 

Table 2

 

Note: Unicode-escape strings will be discussed later.

Even though Java processes strings in Unicode (UTF-8, UCS2), external systems that interface with Java is not necessarily using Unicode and there is a good chance that MBCS is still used there.  Character corruption can take place whenever modules within your Java classes or third-party classes, that are used to communicate with an external system, are not implementing the conversion or are not handling it correctly.  For example, if you are storing data in an external RDBMS in the MBCS format, you might want to verify that the JDBC driver that you use can convert Unicode strings to MBCS properly to avoid data corruption.  This is because the JDBC driver is responsible for converting data passed to the methods of java.sql.Statement class  (e.g. executeUpdate method) in whatever encoding used in the database. Since java.sql.Statement class takes the String data type for a SQL statement, if your database only supports MBCS, JDBC has to be able to convert the data in Unicode to a MBCS correctly.  Since not all the characters supported in Unicode are supported in a MBCS,  in order to guarantee the integrity of data, you also have to make sure that the data that are not supported in the target MBCS will be filtered out before being passed to the JDBC driver.

 

Using javac and locale

Another thing you have to consider is the locale of the machine you run javac on.

Let's think of a situation where two engineers are working on the same source code and both of them use javac to compile the ".java" file.

In this scenario, the first engineer's machine uses the Japanese Windows XP, so javac can handle the source code that contains Shift-JIS characters correctly.  During the compilation, strings in CP932 will be converted to UTF-8 correctly as we discussed earlier.  However, if the second engineer (for example, configuration management engineer) tries to compile the same source file on a machine with the English-US locale, an error will be generated as shown below:

 

Javac uses the locale of the machine in converting from the native character set to UTF-8, and the first character of the Japanese string has 0x5c in the second byte.  Since 0x5c is assigned to the backslash in the ASCII character set, javac misinterprets it as the backslash instead of the second byte of Kanji, and generates an error.

If you remove the first Japanese letter "ソ", the file can be compiled on the machine with the English-US locale.  However, the compiled class file now contains corrupt characters.

When you run the program on the Japanese machine, you can see this problem at offset 4 to 6 as shown below. In a sense, this is harder to detect because there is no error during the compilation.