2

I'am scraping web-page using Jsoup library by selecting class attributes which contains "nav" string in them.

This is the code which fetch HTML of the site:

var bodyString = Jsoup.connect(url)
                .ignoreContentType(true)
                .userAgent("Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:25.0) Gecko/20100101 Firefox/25.0")
                .timeout(12000)
                .followRedirects(true)
                .execute()
                .body();

Example of Html which is selected by Jsoup CSS selector: enter image description here

Yet in browser same website Html looks like this: enter image description here

As you can see ul element with id="varPreviewMenu" contains li elements which HTML retrieved by Jsoup does not contains.

How can I get those elements?

DisappointedByUnaccountableMod
  • 6,656
  • 4
  • 18
  • 22
Romas Augustinavičius
  • 403
  • 2
  • 4
  • 16
  • The first part of this answer applies to your problem https://stackoverflow.com/questions/54505676/jsoup-hidden-div-class/54507805#54507805 – Krystian G May 02 '19 at 20:21
  • Have a look here: https://stackoverflow.com/questions/22390741/jsoup-get-dynamically-generated-html/22400976#22400976 – luksch May 02 '19 at 22:05

1 Answers1

2

Most likely the elements you see are dynamically added to the DOM by some JavaScript code. That means they are not available in the body of the request when you use Jsoup.

Code-Apprentice
  • 81,660
  • 23
  • 145
  • 268