How it is possible to use jodd.http.HttpRequest to load a page content that is generated by javascript?

Question

I try to load a page content with:

HttpResponse response2 = HttpRequest.get(_PAGE_URL).cookies(response.cookies()).send();

In a browser, the page source is full of javascript to generate the DOM, but in the Web Inspector of the browser I can see the generated source.

The question is, can I somehow retrieved the generated page content by Jodd's utilities?

Check my edited answer - you can use Selenium driver, which works quite good. — igr, Jan 25 '22 at 08:53

igr · Accepted Answer · 2022-01-25T08:53:01.423

You can't. You can just download the static HTML content (as you did) and then you would need to render it using some other tool.

Since Java 8, you can use JavaFX's WebView Component (as far as I remember), but please search for other tools as well (maybe cef?)

EDIT See: https://github.com/igr/web-scraper (based on Selenium WebDriver). One thing I miss is better control over request/response.

There is also HtmlUnit, but from the reviews, it seems Selenium is a better choice.

1 Answers1