How to parse a HTML page after waiting for JS to load the content?

Question

I want to get the source code of a HTML page, but it has a JS script that loads the data dynamically and attaches the received content to the page.

I am using jsoup to parse HTML, but it only returns the content before the JS execution, so I do not receive the dynamically loaded data.

How can I get the source code after the script execution using any framework, cli, etc?

something like document.getElementById("elementid").outerHTML after the item is rendered. — Bindrid, Dec 15 '15 at 03:03
Best bet is to run it through Phantom and use Phantom's `page.content`. — , Dec 15 '15 at 04:03
See http://stackoverflow.com/questions/16706777/using-phantom-js-evaluate-how-can-i-get-the-html-of-the-page/16706908#16706908. — , Dec 15 '15 at 04:59
Please specify what framework you're using, share some code, and mention what you tried so far — gafi, Dec 15 '15 at 06:45
You could parse the url where the content is being loaded from — javiercf, Dec 19 '15 at 00:05
JavaScript does not change HTML source. It sounds like you're looking for an HTML dump of the page DOM, as modified by all load-time Javascript. Is this correct? — Ari Fordsham, Sep 09 '20 at 16:44

score 0 · Answer 1 · answered Dec 19 '15 at 09:33

You can use;

$(window).load();

Which will wait until the page is loaded.

Another option:

function whenAvailable(name, callback) {
var interval = 10; // ms
window.setTimeout(function() {
    if (window[name]) {
        callback(window[name]);
    } else {
        window.setTimeout(arguments.callee, interval);
    }
}, interval);
}

And use it like this:

whenAvailable("jsLoadFunction", function(t) {
// do something
});

score -1 · Answer 2 · answered Mar 21 '18 at 06:18

-1

You can use load event of window. Example :

window.onload = function(){
        /*Parse Html*/
}

answered Mar 21 '18 at 06:18

Majar K

34
7

How to parse a HTML page after waiting for JS to load the content?

2 Answers2