Looping over URLs in Casperjs

Question

I'm trying to open a couple of URLs in sequence with Casperjs.

Here is a simplified version of my actual code.

casper = require('casper').create()
casper.start()

casper.then () ->
  items = ['http://www.google.com', 'http://www.yahoo.com']

  for item, idx in items
     this.open item
     this.then ((idx) ->
        () ->
            this.capture idx + '.png')(idx)

casper.run()

In my actual code, the items array is generated at runtime. I expect that this code will provide screenshots of google.com and yahoo.com, but actually both screen shots end up as yahoo.com. I feel like this has something to do with creating a closure inside a loop, but I can't see how. When open is called, item refers to a specific string. I'm never closing over it, am I?

score 1 · Accepted Answer · edited May 23 '17 at 12:05

1

The problem is that casper.open doesn't add a step, so it is not asynchronous (like thenOpen). It is immediately executed inside of the outer then. But the following then is asynchronous so its step is added to be executed after the current step (outer then). Both opens are executed, but the callbacks of both thens are executed after the last open.

So the solution would be to use thenOpen as a combined asynchronous step. There is a slight problem with the straigt forward solution:

CoffeeScript is a little language that compiles into JavaScript.

Which means that JavaScript misconceptions apply. This is the one: JavaScript closure inside loops – simple practical example

The for item, idx in items line is still a for loop in JavaScript. JavaScript has function level scope and casper.then is an asynchronous step function which means that the callbacks of all the casper.then calls are executed after the loop has run completely through (more precisely after the current step is finished or after casper.run was called), but casper.open is immediately executed.

Solution: Combine casper.open and casper.then to casper.thenOpen and pass item and idx through an IIFE so that they are fixed to each iteration:

for item, idx in items
    ((item, idx) ->
        casper.thenOpen item, () ->
            this.capture idx + '.png'
    )(item, idx)

or use Array.prototype.forEach:

items.forEach (item, idx) ->
    casper.thenOpen item, () ->
        this.capture idx + '.png'

edited May 23 '17 at 12:05

Community

1
1

answered Jan 09 '15 at 10:31

Artjom B.

61,146
24
125
222

I'm not that fluent in coffeescript, so I don't know for sure if the above code is syntactically correct. – Artjom B. Jan 09 '15 at 10:32
Doesn't `casper.open` add the open page command to the stack of steps? The actual opening of the page happens after `casper.run` called as well so why would there be a sequencing issue? The reason I put the `idx` inside a closure was as you say, because I want the specific index to be preserved at runtime. But why would a closure be needed for the `casper.open` call? Your idea of bundling them together worked but I hope you understand why I'm still a little confused. Thanks – Mark Jan 09 '15 at 14:13
Yes, I thought that was not that clear on my part. I hope it is better now. – Artjom B. Jan 09 '15 at 14:40
I'm still a little confused. In my mind I translate `casper.open(string)` and `casper.then(func)` to `someQueue.push(["open", string])` and `someQueue.push(["then", func])`. I see why, in the second case, evaluation of the `func` may fail if the names that it closes over have changed, but why would the value of `["open", string]` ever be changed? It's just an array of strings. – Mark Jan 09 '15 at 15:58
1

"`someQueue.push(["open", string])`" doesn't happen, because `casper.open` is not a step function. It is immediately executed, but "`someQueue.push(["then", func])`" does happen and it is executed after the second `casper.open`. So both screenshots show the same page. – Artjom B. Jan 09 '15 at 16:02
Thanks for bearing with me. I must have a fundamental misunderstanding somewhere. How could `open` be immediately executed if execution occurs when `casper.run` is called? Surely `run` must look inside some table to find an `open` command saved. – Mark Jan 09 '15 at 16:05
1

`open` is immediately executed when it is called, but it is only called inside of the outer `then`. It might help if you look into the source code. – Artjom B. Jan 09 '15 at 16:09

Looping over URLs in Casperjs

1 Answers1