file reading encoding trouble

Question

I've a file to read save, do something with its informations and then rewrite them back to another file. the problem is that the original file contains some characters from asian languages like 坂本龍一, 東京事変 and メリー (I guess they're chinese, japanese and korean). I can see them using Notepad++.

the problem is when I read them and write those things via java they get corrupted and I see weird stuff in my output file like ???????? or Жанна БичевÑ?каÑ? I think I got something wrong with the encoding but I've no idea of which to use and how to use it.

can someone help me? here's my code:

    String fileToRead= SONG_2M;
            Scanner scanner = new Scanner(new File(fileToRead), "UTF-8");

            while (scanner.hasNextLine()) {

                String line = scanner.nextLine();
                String[] songData = line.split("\t");
                if (/*something*/) {
                    save the string in the map
                }
            }
            scanner.close();

            saveFile("coded_artist_small2.txt");
}

    public void saveFile(String fileToSave) throws FileNotFoundException, UnsupportedEncodingException {
            PrintWriter writer = new PrintWriter(fileToSave, "UTF-8");

            for (Entry<String, Integer> entry : artistsMap.entrySet()) {
                writer.println(entry.getKey() + DELIMITER + entry.getValue());
            }

            writer.close();
        }

Answer 1

It is likely that your input file is not, in fact, encoded in UTF-8 (an encoding using two bytes per character satisfying the unicode standard). For instance, the character 坂 you are seeing is unicode 0x5742. If, in fact, your file is encoded in ASCII, that should be displayed as character 0x57 followed by 0x42 - i.e. 9*.

If you're unsure of your file's encoding - take a guess that it might be ASCII text. Try removing the encoding when you set up the Scanner i.e. make the second line of your code

Scanner scanner = new Scanner(new File(fileToRead));

If, in fact, you know the file is unicode, there are different encodings. See this answer for a more comprehensive unicode reader - dealing with various unicode encodings.

For your output - you need to decide how you want the file encoded : some unicode encoding (e.g. UTF-8) or as ASCII.