encoding issue mapper.readTree

Question

well I am working on a yugioh game. I tried to get data from api and put it to database (sql server). The issue is when I use map.readTRee() to convert the data to json , It puts ? instead of the real caractere. What is strange is when I read the data stream threw BufferedReader System.out.printlen() read it correctly.

public class ApiEngine {
    
    /**
     * This function will collect all cards data from 'db.ygoprodeck.com/api/v7/cardinfo.php?'
     *@return the result of API research that is a JSonNode type, null otherwise
     * 
     */
    public static JsonNode getCardData()
    {
        try 
        {
             URL url = new URL("https://db.ygoprodeck.com/api/v7/cardinfo.php?name=astral kuriboh");
    
             HttpURLConnection conn = (HttpURLConnection) url.openConnection();
             
             conn.setRequestMethod("GET");
             
             conn.connect();
    
             //Check if connect is made
             int responseCode = conn.getResponseCode();
    
             
             if (responseCode != 200) 
             {
                 throw new RuntimeException("HttpResponseCode: " + responseCode);
             } 
             else 
             {
                BufferedReader br = new BufferedReader(new InputStreamReader( url.openStream(), StandardCharsets.UTF_8));    
          

                /*String strCurrentLine;
                
                while ((strCurrentLine = br.readLine()) != null) {
                        System.out.println(strCurrentLine);
                    }*/
                    
       
                ObjectMapper mapper = new ObjectMapper();
                
                JsonNode cardData = mapper.readTree(br);
                
                            
                return cardData;
             }
         } 
        catch (Exception e) 
        {
             e.printStackTrace();
             
             return null;
        }
    }
}

public class ExporterCardDataEngine {
    
    
    
    
    /**
     * This function will export API data to database
     * @throws SQLException 
     */
    public static void exportApiData() throws SQLException
    {
        
        JsonNode cards = ApiEngine.getCardData().get("data");
        
        Connection connexion = DatabaseEngine.connect();
        
        int size = cards.size();
        
        for (int i = 0; i < size; i++)
        {
            System.out.println((i + 1) +"/" + size + cards.get(i).get("desc"));
            
            DatabaseEngine.insertCard(connexion, cards.get(i));
        }
    }
    
    public static void main(String args[])
    {
        try {
            ExporterCardDataEngine.exportApiData();
        } catch (SQLException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
    }

}

Here is the result when printing the bufferedReader

"desc":"You can reveal 1 \"Number\" Xyz Monster in your Extra Deck; Special Summon this card from your hand. If you do, this card's Level becomes the revealed monster's Rank, also while it is face-up in the Monster Zone, you cannot Special Summon monsters from the Extra Deck, except \"Number\" Xyz Monsters. You can only use this effect of \"Astral Kuriboh\" once per turn. A \"Number\" Xyz Monster that was Summoned using this card on the field as material gains this effect.\r\n\u25cf Cannot be destroyed by battle or your opponent's card effects."

Here after the mapper.readTree()

1/1"You can reveal 1 \"Number\" Xyz Monster in your Extra Deck; Special Summon this card from your hand. If you do, this card's Level becomes the revealed monster's Rank, also while it is face-up in the Monster Zone, you cannot Special Summon monsters from the Extra Deck, except \"Number\" Xyz Monsters. You can only use this effect of \"Astral Kuriboh\" once per turn. A \"Number\" Xyz Monster that was Summoned using this card on the field as material gains this effect.\r\n? Cannot be destroyed by battle or your opponent's card effects."

thank you for your help

I don't use Jackson but this might be applicable: [Jackson ObjectMapper with UTF-8 encoding?](https://stackoverflow.com/q/10004241) — 001, Oct 24 '22 at 17:12
well i didn't get any solution, do you use another json thing at it works ? — Lyna DJELMOUDI, Oct 27 '22 at 22:45
Are you sure this is just not a symptom of the console - or whatever you are using to view the text - not being able to print a '●'? Usually when the console can't print a character because of the wrong font/encoding it will print a '?'. What OS are you using? — 001, Oct 28 '22 at 13:55
If using Windows, click the button on the top left of the console window and choose "Properties". Then change the font to "Consolas". Then okay that and go back to the command prompt. Type "CHCP 65001" at the command prompt to change the code page. Then run your code. — 001, Oct 28 '22 at 14:03
Johnny Mopp thank you for the tip I will try it and tell you if it worked. But if it is the console issue, why with the bufferedreader it is well printed \n\r\u..., and after the mapping not ? — Lyna DJELMOUDI, Oct 29 '22 at 16:45
`\u25cf` is the unicode sequence for '●'. `BufferedReader` is not processing it but just leaving it as text. The JSON parser then converts the code point to an actual character. Then when you try to print the '●' char in a console that doesn't support it you get `?`. — 001, Oct 31 '22 at 12:41
okay tha,k you for yout help, I fotget to answer, I'am using eclipse as ide and I am on a windows system — Lyna DJELMOUDI, Nov 01 '22 at 21:10

score 0 · Answer 1 · edited Nov 09 '22 at 03:50

So after research and some manipulations in my eclipse IDE. I finally come up with a solution that worked for me. Hope it will help :).

First thing I tried to change some default encoding in eclipse : a) Window > Preferences > General > Content Types, set UTF-8 as the default encoding for all content types.

b) Window > Preferences > General > Workspace, set Text file encoding to Other: UTF-8 c) Run > RunConfiguration -> Common -> Encoding -> set Other to UTF-8
Welle the first step didn't work for me so I tried this code

for (int i = 0; i < size; i++)
{
    Charset utf8 = Charset.forName("UTF-8");
    Charset def = Charset.defaultCharset();

    byte[] bytes = cards.get(i).get("desc").asText().getBytes("UTF-8");
    
    String message = new String(bytes , def.name());

    PrintStream printStream = new PrintStream(System.out, true, utf8.name());
    
    printStream.println(message); // should print your character
    
}

And here is what I have in the console :) :) :)

● Cannot be destroyed by battle or your opponent's card effects.

: Well... You can find everything here

encoding issue mapper.readTree

1 Answers1