Decode html response

Question

I am trying to scrape some data from following web page: https://bitcoin.pl/ I am getting a response from server and extract body. I want to extract links from body. However, can't do it because body has not been decoded properly and contains escape characters.

I tried some solutions from the following:
How to unescape HTML character entities in Java?
https://howtodoinjava.com/java/string/unescape-html-to-string/

Below I provide code I wrote:

import com.mashape.unirest.http.HttpResponse;
import com.mashape.unirest.http.Unirest;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;

public class test_scraper {

    public static void main(String[] args) throws Exception {
        final String USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36";
        Unirest.setDefaultHeader("User-Agent",USER_AGENT);

        final HttpResponse<String> response = Unirest.post("https://bitcoin.pl/?ajax-request=jnews")
                .header("Accept","application/json, text/javascript, */*; q=0.01")
                .header("Content-Type","application/x-www-form-urlencoded; charset=UTF-8")
                .header("Referer","https://bitcoin.pl/")
                .header("user-agent",USER_AGENT)
                .header("accept-language", "en-US,en;q=0.9")
                .header("X-Requested-With","XMLHttpRequest")
                .header("accept-encoding:", "gzip, deflate, br")

                .queryString("lang","pl_PL")
                .queryString("action","jnews_module_ajax_jnews_block_5")
                .queryString("data[current_page]",2)
                .queryString("data[attribute][number_post]", 1)
                .asString();

        //System.out.println(response.getHeaders());
        //System.out.println(response.getBody());
        final Document html = Jsoup.parseBodyFragment(response.getBody());
        System.out.println(Jsoup.parse(response.getBody()));

        }
    }

I receive an exact same response like from the browser(inspector mode -> Network -> XHR -> Response) however I want to get HTML from Preview which is already decoded.

What I receive is (some part of response):

<html>
 <head></head>
 <body>
  {"content":"
  <div class="\&quot;jeg_posts" jeg_load_more_flag\">
   \n 
   <article class="\&quot;jeg_post" jeg_pl_lg_2 post-9771 post type-post status-publish format-standard has-post-thumbnail hentry category-kryptowaluty tag-bitcoin tag-chinski-bank-ludowy tag-chinski-banki-centralny tag-chiny tag-cyfrowa-waluta tag-libra tag-token-pboc\">
    \n 
    <div class="\&quot;jeg_thumb\&quot;">
     \n \n 
     <a href="\&quot;https:\/\/bitcoin.pl\/chiny-data-emisji-waluty\/\&quot;"></a>
     <div class="\&quot;thumbnail-container" animate-lazy size-715 \">
      <a href="\&quot;https:\/\/bitcoin.pl\/chiny-data-emisji-waluty\/\&quot;"><img width="\&quot;350\&quot;" height="\&quot;250\&quot;" src="\&quot;https:\/\/bitcoin.pl\/wp-content\/themes\/jnews\/assets\/img\/jeg-empty.png\&quot;" class="\&quot;attachment-jnews-350x250" size-jnews-350x250 lazyload wp-post-image\" alt="\&quot;chiny\&quot;" data-src="\&quot;https:\/\/bitcoin.pl\/wp-content\/uploads\/2019\/09\/chiny-350x250.jpg\&quot;" data-sizes="\&quot;auto\&quot;" data-srcset="\&quot;https:\/\/bitcoin.pl\/wp-content\/uploads\/2019\/09\/chiny-350x250.jpg" 350w, https:\ \ bitcoin.pl\ wp-content\ uploads\ 2019\ 09\ chiny-120x86.jpg 120w, chiny-750x536.jpg 750w\" data-expand="\&quot;700\&quot;" data-animate="\&quot;0\&quot;">&lt;\/div&gt;&lt;\/a&gt;\n 
       <div class="\&quot;jeg_post_category\&quot;">
        \n 
        <span><a href="\&quot;https:\/\/bitcoin.pl\/category\/kryptowaluty\/\&quot;" class="\&quot;category-kryptowaluty\&quot;">Kryptowaluty&lt;\/a&gt;&lt;\/span&gt;\n &lt;\/div&gt;\n &lt;\/div&gt;\n </a>

How to decode above properly to get correct HTML?

mentallurg · Accepted Answer · 2019-09-26T01:28:39.133

0

This service returns a JSON:

{
    "content": "<div class=...",
    "next": false,
    "prev": true
}

Jsoup is not needed, because this HTML is embedded into a JSON object. Use Jackson instead:

ObjectMapper mapper = new ObjectMapper();
Map map = mapper.readValue(body, Map.class);
String content = map.get("content").toString();
System.out.println(content);

And you will get normal HTML without any escaping:

<div class="jeg_posts jeg_load_more_flag">
    <article class="jeg_post ...
        <div class="jeg_thumb">
            ...

The class ObjectMapper above is com.fasterxml.jackson.databind.ObjectMapper, don't confuse it with similar class from Unirest.

To use Jackson, add following dependency to your Gradle file, in Maven it is similar:

implementation 'com.fasterxml.jackson.core:jackson-databind:2.10.0.pr3'

edited Sep 26 '19 at 01:28

answered Sep 24 '19 at 18:53

mentallurg

4,967
5
28
36

thanks @mentallurg I tried this: `String my_html = org.jsoup.parser.Parser.unescapeEntities(response.getBody(),false);` It returns same string. – Rafal Sep 25 '19 at 06:06
@Rafal: If the answer is helpful, you can show that by upvoting. – mentallurg Oct 03 '19 at 08:42
I upvoted but have not enough reputation. My vote was recorded but won't be visible on the page. – Rafal Oct 04 '19 at 09:10

Decode html response

1 Answers1