What is causing this ASCII characters to become question marks?

Question

I have this Java code where a config.yaml should get generated with a custom header, which contains various ASCII characters, such as ASCII-219. Homewer when i compile the plugin and look into the config.yml, the characters get just replaced with question marks. I've already looked into my IntelliJ IDEA file encoding and it's set to UTF8.

I've tried to use UTF-8 with BOM

        FileConfiguration config = main.plugin.getConfig();

        getConfig().options().header(
                        "  █████▒▄▄▄       ██▓     ██▓    ▓█████ ███▄    █   ██████  █    ██  ██▀███   ██▒   █▓ ██▓ ██▒   █▓ ▒█████   ██▀███                          " + "\n" +
                        " ▓██   ▒▒████▄    ▓██▒    ▓██▒    ▓█   ▀ ██ ▀█   █ ▒██    ▒  ██  ▓██▒▓██ ▒ ██▒▓██░   █▒▓██▒▓██░   █▒▒██▒  ██▒▓██ ▒ ██▒               " + "\n" +
                        " ▒████ ░▒██  ▀█▄  ▒██░    ▒██░    ▒███  ▓██  ▀█ ██▒░ ▓██▄   ▓██  ▒██░▓██ ░▄█ ▒ ▓██  █▒░▒██▒ ▓██  █▒░▒██░  ██▒▓██ ░▄█ ▒          " + "\n" +
                        " ░▓█▒  ░░██▄▄▄▄██ ▒██░    ▒██░    ▒▓█  ▄▓██▒  ▐▌██▒  ▒   ██▒▓▓█  ░██░▒██▀▀█▄    ▒██ █░░░██░  ▒██ █░░▒██   ██░▒██▀▀█▄            " + "\n" +
                        " ░▒█░    ▓█   ▓██▒░██████▒░██████▒░▒████▒██░   ▓██░▒██████▒▒▒▒█████▓ ░██▓ ▒██▒   ▒▀█░  ░██░   ▒▀█░  ░ ████▓▒░░██▓ ▒██▒  " + "\n" +
                        "▒ ░    ▒▒   ▓▒█░░ ▒░▓  ░░ ▒░▓  ░░░ ▒░ ░ ▒░   ▒ ▒ ▒ ▒▓▒ ▒ ░░▒▓▒ ▒ ▒ ░ ▒▓ ░▒▓░   ░ ▐░  ░▓     ░ ▐░  ░ ▒░▒░▒░ ░ ▒▓ ░▒▓░               " + "\n" +
                        "░       ▒   ▒▒ ░░ ░ ▒  ░░ ░ ▒  ░ ░ ░  ░ ░░   ░ ▒░░ ░▒  ░ ░░░▒░ ░ ░   ░▒ ░ ▒░   ░ ░░   ▒ ░   ░ ░░    ░ ▒ ▒░   ░▒ ░ ▒░                        " + "\n" +
                        "░ ░     ░   ▒     ░ ░     ░ ░      ░     ░   ░ ░ ░  ░  ░   ░░░ ░ ░   ░░   ░      ░░   ▒ ░     ░░  ░ ░ ░ ▒    ░░   ░                                      " + "\n" +
                        "░  ░    ░  ░    ░  ░   ░  ░        ░       ░     ░        ░           ░   ░        ░      ░ ░     ░                                                                   " + "\n" +
                        "░            ░                                                                                                                                                                    " + "\n" +
                        "Plugin BETA 1.0.0/                                                                                                                                                                  " + "\n" +
                        "FallenSurvivor is made by FendiTony777, all rights are reserved to him.                                                                                                            " + "\n" +
                        "You can contact me from the discord link provided.                                                                                                                                   " + "\n" +
                        "---------------------------------------------                                                                                                                                        " + "\n"
        );```

What do you mean by "ASCII-219"? ASCII characters are in the range 0 to127 only. Also, I see no question marks in what you posted. — , Aug 04 '19 at 13:14
0x219 is also Û. The box drawing characters you're using are [much further out](https://www.unicode.org/charts/PDF/U2580.pdf) — Draco18s no longer trusts SE, Aug 04 '19 at 14:02

score 0 · Answer 1 · answered Aug 04 '19 at 14:50

You need to be aware of the character encoding of your file and make sure that all tools in the processing chain honor it correctly. As already noted in comments, ASCII covers only the 7-bit subset of unicode code points (or rather, unicode was defined to include ASCII in the range 0-127). These block graphic characters are not ASCII.

It also looks like you combined glyphs that are not defined to work together seamlessly (this is also noted on the Wikipedia page https://en.m.wikipedia.org/wiki/Block_Elements.) The space glyph is a little narrower than the blocks, and the full blocks look a little higher than the stippled blocks, but this probably does not matter much depending on the font you'll be using for rendering.

If you have no control over the FileConfiguration class which probably doesn't use an utf-8 encoding for writing, you're somewhat out of luck unless the generation of config.yml is a one-time process. In that case, you can edit the generated file manually and replace the mis-encoded strings with the correct ones.

However, you then still need to make sure that the functions which read your yaml file work correctly and preserve the utf-8 codes. If config file generation and config file reading is done by the same class it's likely that reading will have similar issues.

What is causing this ASCII characters to become question marks?

1 Answers1