File.list() retrieves file names with NON-ASCII characters incorrectly on Mac OS X when using Java 7 from Oracle

Question

I have a problem using File.list() with file names with NON-ASCII characters incorrectly retrieved on Mac OS X when using Java 7 from Oracle.

I am using the following example:

import java.io.*;
import java.util.*;

public class ListFiles {

  public static void main(String[] args) 
  {
    try { 
      File folder = new File(".");
      String[] listOfFiles = folder.list(); 
      for (int i = 0; i < listOfFiles.length; i++) 
      {
        System.out.println(listOfFiles[i]);
      }
      Map<String, String> env = System.getenv();
      for (String envName : env.keySet()) {
        System.out.format("%s=%s%n",
            envName,
            env.get(envName));
      }
    } catch (Exception e) { 
      e.printStackTrace(); 
    } 
  }

}

Running this example with Java 6 from Apple, everything is fine:

....
Folder-ÄÖÜäöüß
吃饭.txt
....

Running this example with Java 7 from Oracle, the result is as follows:

....
Folder-A��O��U��a��o��u����
������.txt
....

But, if I set the environment as follows (not set in the two cases above):

LANG=en_US.UTF-8

the result with Java 7 from Oracle is as expected:

....
Folder-ÄÖÜäöüß
吃饭.txt
....

My problem is that I don't want to set the LANG environment variable. It's a GUI application that I want to deploy as an Mac OS X application, and doing so, the LSEnvironment setting

<key>LSEnvironment</key>
<dict>
  <key>LANG</key>
  <string>en_US.UTF-8</string>
</dict>

in Info.plist takes no effect (see also here)

What can I do to retrieve the file names correctly in Java 7 from Oracle on Mac OS X without having to set the LANG environment? In Windows and Linux, this problem does not exist.

EDIT:

If I print the individual bytes with:

byte[] x = listOfFiles[i].getBytes();
for (int j = 0; j < x.length; j++) 
{
    System.out.format("%02X",x[j]);
    System.out.print(" ");
}
System.out.println();

the correct results are:

Folder-ÄÖÜäöüß
46 6F 6C 64 65 72 2D 41 CC 88 4F CC 88 55 CC 88 61 CC 88 6F CC 
88 75 CC 88 C3 9F 
吃饭.txt
E5 90 83 E9 A5 AD 2E 74 78 74 

and the wrong results are:

Folder-A��O��U��a��o��u����
46 6F 6C 64 65 72 2D 41 EF BF BD EF BF BD 4F EF BF BD EF BF BD 
55 EF BF BD EF BF BD 61 EF BF BD EF BF BD 6F EF BF BD EF BF BD 
75 EF BF BD EF BF BD EF BF BD EF BF BD  
������.txt
EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD 2E 74 78 74 

So one can see that Files.list() replaces some bytes with UTF-8 "EF BF BD" = Unicode U+FFFD = Replacement Character, if LANG is not set (only Java 7 from Oracle).

Answer 1

If everything else fails, create a wrapper for the JVM that sets the LC_CTYPE environment variable and then launches your application. OS X doesn't care which program the plist tells it to run does it? It's probably simplest to create this wrapper in shell script:

#!/bin/bash
export LC_CTYPE="UTF-8" # Try other options if this doesn't work
exec java your.program.Here

The problem is with the way Java - any version of Java, from either Apple or Oracle - reads the names of files from the file system. Names of files on the file system are essentially binary data, and they must be decoded in order to use them as String in Java. (You can read more about this issue in my blog.)

The detection of the encoding varies from platform to platform and version to version, so this must be where Apple Java 6 and Oracle Java 7 differ: Java 6 detects correctly that the system is set to UTF-8, while Java 7 gets it wrong.

Strangely though, when I try to reproduce the issue with the following program I find that both Java 6 and Java 7 correctly use UTF-8 to decode file names (they are printed correctly to the terminal). For other I/O, Java 6u35 is using MacRoman as the default charset, while Java 7u7 uses UTF-8 (shown by the file.encoding system property).

import java.io.*;

public class Test {
  public static void main(String[] args) {
    System.setOut(new PrintStream(System.out, true, "UTF-8"));
    System.out.println(System.getProperty("file.encoding"));
    for (File f: new File(".").listFiles) {
      System.out.println(g.getName());
    }
  }
}

When I run locale on OS 10.7 I get this output. It seems that on my system Java 6 doesn't interpret correctly the value given for LC_CTYPE. As far as I know the system has no customizations and everything is set to English, so this should be the default configuration:

LANG=
LC_COLLATE="C"
LC_CTYPE="UTF-8"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=

Answer 2

Since running from Java6 gives correct result, would this:

System.out.println(new String(listOfFiles[i].getBytes(),"UTF-8"));

solve the problem?

This suggested constructor explicitly interprets the listOfFiles[i] string as an UTF-8 encoded string.

EDIT:

As it is not working it means that UTF-8 is not the default encoding for os x. Wikipedia says that Mac OS Roman is, though. So I d suggest to try:

System.out.println(new String(listOfFiles[i].getBytes(),"MacRoman"));

but that should be the same as

System.out.println(new String(listOfFiles[i].getBytes()));

So if that is not working also, that leads to conclusion that it might be a bug as Andrew Thomson stated in comment to your question.

Answer 3

It's a known bug in OpenJDK. OS X 10.6 and OS X 10.7 return different values for the default locale. See bug http://java.net/jira/browse/MACOSX_PORT-204 and http://java.net/jira/browse/MACOSX_PORT-165. If you're having this problem, vote for getting it fixed.

Answer 4

Downgrade your JDK to the built in Mac OSX JDK. If you do, the problem should vanish.

In addition, you may also want to set your run configuration in Eclipse to run as UTF-8.

Answer 5

It's a bug in the old java File api (maybe just on a mac). Anyway, it's all fixed in the new java.nio.

I have several files containing unicode characters in the filename and content that failed to load using java.io.File and related classes. After converting all my code to use java.nio.Path EVERYTHING started working. And I replaced org.apache.commons.io.FileUtils (which has the same problem) with java.nio.Files...

...and be sure to read and write the content of file using an appropriate charset, for example: Files.readAllLines(myPath, StandardCharsets.UTF_8)