How to do nested looping over many pages in CasperJS

Question

I don't have a clue where to start with this. Basically I need CasperJS to run through about 15 different pages, each page that it runs through it needs to get the data for 150 different locations that need to be set as cookie values. For each location, I need to check the data for 5 different dates.

Any one of these seems pretty straight forward, but trying to get all three to happen is confusing me.

I tried to set it up this way:

for(Iterate through URLs){
  for(Iterate through locations){
    for(Iterate through dates){
      phantom.addCookie({
        // Cookie data here based on location and date
      });
      casper.start(url)
      .then(function(){
        // Do some stuff here
      })
      .run();
    }
  }
}

Essentially what it does is loop through everything, then load the page based on the last link, at the last location, on last date. But every other location gets skipped. Is there an easier way to do this? Perhaps better, is there a way to tell my JavaScript loop to wait for casper to finish doing what it needs to do before jumping to the next loop iteration?

I'm happy to provide more details if needed. I tried to simplify the process as best I can without cutting out needed info.

Answer 1

That's pretty much it. Two things to look out for:

• casper.start() and casper.run() should only be called once per script. You can use casper.thenOpen() to open different URLs.

• Keep in mind that all casper.then*() and casper.wait*() functions are asynchronous step functions and are only scheduled for execution after the current step. Since JavaScript has function level scope, you need to "fix" the iteration variables for each iteration otherwise you will get only the last URL. (More information)

Example code:

casper.start(); // deliberately empty

for (var url in urls) {
    for (var location in locations) {
        for (var date in dates) {
            (function(url, location, date){
                casper.then(function(){
                    phantom.addCookie({
                        // Cookie data here based on location and date
                    });
                }).thenOpen(url)
                    .then(function(){
                        // Do some stuff here
                    });
            })(url, location, date);
        }
    }
}

casper.run(); // start all the scheduled steps

If you use Array.prototype.forEach instead of the for-loop, then you can safely skip the use of the IIFE to fix the variables.

I'm not sure, but you may need to first open a page to then add a cookie for that domain. It may be possible that PhantomJS only accepts a cookie when that domain for that cookie is currently open.