Read unicode text files with java

Question

Real simple question really. I need to read a Unicode text file in a Java program.

I am used to using plain ASCII text with a BufferedReader FileReader combo which is obviously not working :(

I know that I can read a String in the 'traditional' way using a Buffered Reader and then convert it using something like:

temp = new String(temp.getBytes(), "UTF-16");

But is there a way to wrap the Reader in a 'Converter'?

EDIT: the file starts with FF FE

score 18 · Accepted Answer · answered Jun 11 '09 at 08:27

18

you wouldn't wrap the Reader, instead you would wrap the stream using an InputStreamReader. You could then wrap that with your BufferedReader that you currently use

BufferedReader in = new BufferedReader(new InputStreamReader(stream, encoding));

answered Jun 11 '09 at 08:27

objects

8,637
4
30
38

2

I want to read Hebrew letters, what would I replace with "encoding"? – CodyBugstein Oct 30 '13 at 16:12
2

to answer my own question, it's "UTF-8" – CodyBugstein Oct 30 '13 at 16:15
'The constructor BufferedReader(InputStreamReader) is undefined'? – Squareoot Dec 15 '16 at 15:29

score 10 · Answer 2 · edited Jun 02 '20 at 19:13

10

Check https://docs.oracle.com/javase/1.5.0/docs/api/java/io/InputStreamReader.html.

I would read source file with something like:

Reader in = new InputStreamReader(new FileInputStream("file"), "UTF-8"));

edited Jun 02 '20 at 19:13

fospathi

537
1
6
7

answered Jun 11 '09 at 08:23

Macarse

91,829
44
175
230

score 7 · Answer 3 · answered Jun 11 '09 at 09:06

Some notes:

the "UTF-16" encoding can read either little- or big-endian encoded files marked with a BOM; see here for a list of Java 6 encodings; it is not explicitly stated what endianness will be used when writing using "UTF-16" - it appears to be big-endian - so you might want to use "UnicodeLittle" when saving the data
be careful when using String class encode/decode methods, especially with a marked variable-width encoding like UTF-16 - use them only on whole data
as others have said, it is often best to read character data by wrapping your InputStream with an InputStreamReader; you can concatenate your input into a single String using a StringBuilder or similar buffer.

Thanks for the link to the encoding types. I found the right one for me. — Roger C S Wernersson, Mar 26 '10 at 07:34

score 2 · Answer 4 · edited May 23 '17 at 11:46

2

I would recommend to use UnicodeReader from Google Data API, see this answer for a similar question. It will automatically detect encoding from the Byte order mark (BOM).

You may also consider BOMInputStream in Apache Commons IO which does basically the same but does not cover all alternative versions of BOM.

edited May 23 '17 at 11:46

Community

1
1

answered Oct 22 '15 at 11:13

stenix

3,068
2
19
30

score 0 · Answer 5 · answered Nov 02 '18 at 08:44

I just had to add "UTF-8" to the creation of the InputStreamReader and special characters could be seen inmediately.

InputStreamReader istreamReader = new InputStreamReader(inputStream,"UTF-8");
BufferedReader bufferedReader = new BufferedReader(istreamReader);

score -1 · Answer 6 · edited May 19 '14 at 18:14

-1

 Scanner scan = new Scanner(new File("C:\\Users\\daniel\\Desktop\\Corpus.txt"));
   while(scan.hasNext()){

   System.out.println(scan.nextLine());
    }

edited May 19 '14 at 18:14

BradleyDotNET

60,462
10
96
117

answered May 19 '14 at 17:55

daniel molla

1

1

Is the Scanner class specific to unicode? Just reading the code (and not being aware of such things) it is difficult to ascertain if this actually answers the question. For issues where the OP may need some conceptual understanding as well as code, it is useful to include a short text description of *why* the code works in your answer. Such a description would be beneficial here. Also, I have edited your post to put the code in "Code Markup" Please do the same in the future as it makes it much easier to read. Welcome to StackOverflow! – BradleyDotNET May 19 '14 at 18:15

score -1 · Answer 7 · edited Jun 13 '18 at 14:34

-1

String s = new String(Files.readAllBytes(Paths.get("file.txt")),"UTF-8");

edited Jun 13 '18 at 14:34

anothernode

5,100
13
43
62

answered Jun 13 '18 at 13:42

aldo

1

Read unicode text files with java

7 Answers7

Linked