Reading strange unicode character in Java?

Question

I have the following text file:

The file was saved with utf-8 encoding.

I used the following code to read the content of the file:

FileReader fr = new FileReader("f.txt");
BufferedReader br = new BufferedReader(fr);
String s1 = br.readLine();
String s2 = br.readLine();
System.out.println("s1 = " + s1.length());
System.out.println("s2 = " + s2.length());

the output:

s1 = 5

s2 = 4

Then I tried to use s1.charAt(0); to get the first character of s1 and it was '' (blank) character. That's why s1 has the length of 5. Even if I tried to use s1.trim(); its length still 5. I dont know why that happened? It worked correctly if the file was saved with ASCII encoding.

Answer 1

Notepad apparently saved the file with a byte order mark, a nonprintable character at the beginning that just marks it as UTF-8 but is not required (and indeed not recommended) to use. You can ignore or remove it; other text editors often give you the choice of using UTF-8 with or without a BOM.

Answer 2

That's actually not a blank character, it's a BOM - Byte Order Mark. Windows uses the BOM to mark files as unicode (UTF-8, UTF-16 and UTF-32) encoded files.

I think you can save the files without the BOM even in Notepad (it's not required actually).

Answer 3

Well, you may be trying to read your file using a different encoding.

You need to use the OutputStreamReader class as the reader parameter for your BufferedReader. It does accept an encoding. Review Java Docs for it.

Somewhat like this:

BufeferedReader out = new BufferedReader(new OutputStreamReader(new FileInputStream("jedis.txt),"UTF-8")))

Or you can set the current system encoding with the system property file.encoding to UTF-8.

java -Dfile.encoding=UTF-8 com.jediacademy.Runner arg1 arg2 ...

You may also set it as a system property at runtime with System.setProperty(...) if you only need it for this specific file, but in a case like this I think I would prefer the OutputStreamWriter.

By setting the system property you can use FileReader and expect that it will use UTF-8 as the default encoding for your files. In this case for all the files that you read and write.

If you intend to detect decoding errors in your file you would be forced to use the OutputStreamReader approach and use the constructor that receives an decoder.

Somewhat like

CharsetDecoder decoder = Charset.forName("UTF-8").newDecoder();
decoder.onMalformedInput(CodingErrorAction.REPORT);
decoder.onUnmappableCharacter(CodingErrorAction.REPORT);
BufeferedReader out = new BufferedReader(new InputStreamReader(new FileInputStream("jedis.txt),decoder));

You may choose between actions IGNORE | REPLACE | REPORT

Answer 4

the null character, for example. when you use (char)0, is translated to ''

It might be that filereader is reading in a null character at the start of the file. I'm not sure why though...

Answer 5

Even if I tried to use s1.trim(); its length still 5.

I expect that you are doing this:

    s1.trim();

That doesn't do what you want it to do. Java Strings are immutable, and the trim() method is creating a new String ... which you are then throwing away. You need to do this:

    s1 = s1.trim();

... which assigns the reference to the new String created by trim() to something so that you can use it.

(Note: trim() doesn't always create a new String. If the original string has no leading or trailing whitespace, the trim() method simply returns it as-is.)