0

I am new to javascript and I am trying to capture an html file mostly generated by javascript code.

The site I want to scrape is this link:

http://www.modcloth.com/shop/dresses/luck-be-a-lady-dress-in-black-and-red

For this specific webpage, I would like to capture all the output after clicking the button named viewall.

The code below is able to capture the current display of the webpage, but it does not seem to capture the display after clicking the viewall button. I have been banging my head against this problem for a couple of days now, and was hoping someone might have a better idea.

// scrape_techstars.js

var webPage = require('webpage');
var page = webPage.create();

var fs = require('fs');
var path = 'modcloth.html'

page.open('http://www.modcloth.com/shop/dresses/luck-be-a-lady-dress-in-black-and-red', function() {
  page.includeJs("http://ajax.googleapis.com/ajax/libs/jquery/1.6.1/jquery.min.js", function() {
    page.evaluate(function() {
      $("#viewall").click();
    });
    var content = page.content;
    fs.write(path,content,'w')
    phantom.exit()
  });
});
Mike.Gahan
  • 4,565
  • 23
  • 39
  • The other question doesn't incorporate the button interaction, which I think might be causing me issues. – Mike.Gahan Jun 02 '16 at 15:13
  • Have you tried to surround `var content = page.content; fs.write(path,content,'w') phantom.exit()` with `setTimeout(function(){ ... }, 5000);`? – Artjom B. Jun 02 '16 at 15:15
  • @Artjom B. ... you are a lifesaver. Thank you so much. – Mike.Gahan Jun 02 '16 at 15:20
  • That's basically what the highest-voted answer in the linked duplicate says, but the 3rd and 4th answers are more elaborate. – Artjom B. Jun 02 '16 at 15:24

0 Answers0