characters not appearing when I print when I import a file?

Question

I'm importing a file into my code and trying to print it. the file contains

 i don't like cake.
 pizza is good.
 i don’t like "cookies" to.
 17.
 29.

the second dont has a "right single quotation" and when I print it the output is

 don�t 

the question mark is printed out a blank square. is there a way to convert it to a regular apostrophe? EDIT:

     public class Somethingsomething {


     public static void main(String[] args) throws FileNotFoundException, 
    IOException {
    ArrayList<String> list = new ArrayList<String>();
    File file = new File("D:\\project1Test.txt");//D:\\project1Test.txt
    if(file.exists()){//checks if file exist

    FileInputStream fileStream = new FileInputStream(file);
    InputStreamReader input = new InputStreamReader(fileStream);
    BufferedReader reader = new BufferedReader(input);      

    String line;
    while( (line = reader.readLine()) != null) { 
     list.add(line);
     }


    for(int i = 0; i < list.size(); i ++){
        System.out.println(list.get(i));
    }

    }

    }}

it should print as normal but the second "don't" has a white block on the apostrophe this is the file I'm using https://www.mediafire.com/file/8rk7nwilpj7rn7s/project1Test.txt edit: if it helps even more my the full document where the character is found here https://www.nytimes.com/2018/03/25/business/economy/labor-professionals.html

Answer 1

It’s all about character encoding. The way characters are represented isn't always the same and they tend to get misinterpreted.

Characters are usually stored as numbers that depend on the encoding standard (and there are so many of them). For example in ASCII, "a" is 97, and in UTF-8 it's 61.

Now when you see funny characters such as the question mark (called replacement character) in this case, it's usually that an encoding standard is being misinterpreted as another standard, and the replacement character is used to replace the unknown or misinterpreted character.

To fix your problem you need to tell your reader to read your file using a specific character encoding, say SOME-CHARSET.

Replace this:

InputStreamReader input = new InputStreamReader(fileStream);

with this:

InputStreamReader input = new InputStreamReader(fileStream, "SOME-CHARSET");

A list of charsets is available here. Unfortunately, you might want to go through them one by one. A short list of most common ones could be found here.

Answer 2

Your problem is almost certainly the encoding scheme you are using. You can read a file in most any encoding scheme you want. Just tell Java how your input was encoded. UTF-8 is common on Linux. Windows native is CP-1250.

This is the sort of problem you have all the time if you are processing files created on a different OS.

See here and Here

Answer 3

I'll give you a different approach...

Use the appropriate means for reading plain text files. Try this:

public static String getTxtContent(String path)
{
    try(BufferedReader br = new BufferedReader(new FileReader(path))) 
    {
        StringBuilder sb = new StringBuilder();
        String line = br.readLine();

        while (line != null) {
        sb.append(line);
        sb.append(System.lineSeparator());
        line = br.readLine();
    }
    return sb.toString(); 
    }catch(IOException fex){ return null; }
}