Removing non-UTF-8 characters from large txt file

Question

I'm working on a 1 Gigabyte JSON text file which I'm trying to parse using Java. However, the parser throws an exception because it runs into the character 'ñ' generating this exception:

Exception Invalid UTF-8 start byte 0x96

I've tried to remove the character using sed and perl, but it seems that they cannot read the character and thus the file remains unchanged. I'd like to remove the character from the whole file or replace it with any other character or string so that the parsing works.

To add, use iso-8859-1 as encoding to specify latin1 in java — jontro, Jun 19 '12 at 16:42

Denys Séguret · Accepted Answer · 2012-06-26T08:09:25.843

5

Your file is not encoded in UTF-8.

You should find the encoding and use this encoding to read the File using InputStreamReader. And then save it if needed in UTF-8 (using for exemple an OutputStreamWriter).

If you don't know the encoding, I suggest you test with a few probable encodings : see Charsets.

edited Jun 26 '12 at 08:09

answered Jun 19 '12 at 16:30

Denys Séguret

372,613
87
782
758

score 2 · Answer 2 · edited Jun 20 '20 at 09:12

2

Yes it may not be UTF-8 see here for some information on how to check what encoding it is: Java : How to determine the correct charset encoding of a stream

with the best answering seeming to point towards the InputStreamReader#getEncoding()

edited Jun 20 '20 at 09:12

Community

1
1

answered Jun 19 '12 at 16:31

David Kroukamp

36,155
13
81
138

Removing non-UTF-8 characters from large txt file

2 Answers2