nodeJS request package load with javascript

Question

I am trying to fetch an instagram profile with request package but since Instagram is loading things via javascript (reactJS) many informations are missing :/

Do you have any idea how to request the page, wait like 5 Seconds and then download it!

Answer 1

Use a headless browser such as PhantomJS.

A module for integrating with Node.JS is available: https://github.com/amir20/phantomjs-node

var phantom = require('phantom');
phantom.create().then(function(ph) {
    ph.createPage().then(function(page) {
         page.open('https://www.instagram.com/collectnet/').then(function(status) {
            console.log(status);
            page.property('content').then(function(content) {
                console.log(content);
                page.close();
                ph.exit();
            });
        });
    });
});

It is hard to detect the end of Javascript execution on a webpage, however if you know what content you need to wait for to load, then you can check the existance of that specific element. An example is demonstrated here: https://stackoverflow.com/a/38149362/2128499