CasperJS Load next page in loop

Question

I've been working on a script which collates the scores for a list of user from a website. One problem is though, I'm trying to load the next page in the while loop, but the function is not being loaded...

casper.then(function () {
    var fs = require('fs');
    json = require('usernames.json');

    var length = json.username.length;
    leaderboard = {};
    for (var ii = 0; ii < length; ii++) {
        var currentName = json.username[ii];
        this.thenOpen("http://www.url.com?ul=" + currentName + "&sortdir=desc&sort=lastfound", function (id) {
                return function () {
                    this.capture("Screenshots/" + json.username[id] + ".png");
                    if (!casper.exists(x("//*[contains(text(), 'That username does not exist in the system')]"))) {

                        if (casper.exists(x('//*[@id="ctl00_ContentBody_ResultsPanel"]/table[2]'))) {
                            this.thenEvaluate(tgsagc.tagNextLink);

                            tgsagc.cacheCount = 0;
                            tgsagc.
                            continue = true;
                            this.echo("------------ " + json.username[id] + " ------------");
                            while (tgsagc.
                                continue) {
                                this.then(function () {
                                    this.evaluate(tgsagc.tagNextLink);
                                    var findDates, pageNumber;
                                    pageNumber = this.evaluate(tgsagc.pageNumber);
                                    findDates = this.evaluate(tgsagc.getFindDates);
                                    this.echo("Found " + findDates.length + " on page " + pageNumber);
                                    tgsagc.checkFinds(findDates);
                                    this.echo(tgsagc.cacheCount + " Caches for " + json.username[id]);
                                    this.echo("Continue? " + tgsagc["continue"]);
                                    this.click("#tgsagc-link-next");


                                });

                            }

                            leaderboard[json.username[id]] = tgsagc.cacheCount;
                            console.log("Final Count: " + leaderboard[json.username[id]]);
                            console.log(JSON.stringify(leaderboard));
                        } else {
                            this.echo("------------ " + json.username[id] + " ------------");
                            this.echo("0 Caches Found");
                            leaderboard[json.username[id]] = 0;
                            console.log(JSON.stringify(leaderboard));

                        }


                    } else {
                        this.echo("------------ " + json.username[id] + " ------------");
                        this.echo("No User found with that Username");
                        leaderboard[json.username[id]] = null;
                        console.log(JSON.stringify(leaderboard));

                    }
                });

Have you checked for errors in the web console? When you say "the function" is not being loaded you mean that the function within then() is not being executed at all, or it is failing when trying to load the subsequent page? — Ben Smith, Nov 13 '13 at 00:10
When it returns the click on the next link, it doesn't load the next page before starting the next iteration of the while loop, let alone at all, I've tried many different combinations, but none work — SquiresSquire, Nov 13 '13 at 00:44
You're doing it wrong. You have to move the core page functionality into a function and use recursion by nesting `casper.then`s. `while` will not work. See my answer [here](http://stackoverflow.com/a/23295645/1816580). — Artjom B., Jun 28 '14 at 22:17

Ben Smith · Answer 1 · 2013-11-13T17:13:23.677

0

while (tgsagc.continue) {
    this.then(function(){
        this.evaluate(tgsagc.tagNextLink);
        var findDates, pageNumber;
        pageNumber = this.evaluate(tgsagc.pageNumber);
        findDates = this.evaluate(tgsagc.getFindDates);
        this.echo("Found " + findDates.length + " on page " + pageNumber);
        tgsagc.checkFinds(findDates);
        this.echo(tgsagc.cacheCount + " Caches for " + json.username[id]);
        this.echo("Continue? " + tgsagc["continue"]);
        return this.click("#tgsagc-link-next");
    });
}

Ok, looking at this code I can suggest a couple of changes you should make:

I don't think you should be calling return from within your function within then(). This maybe terminating the function prematurely. Looking at the casperjs documentation, the examples don't return anything either.
Within your while loop, what sets "tgsagc.continue" to false?
Don't use "continue" as a variable name. It is a reserved word in Javascript used for terminating an iteration of a loop. In your case this shouldn't be a problem, but its bad practice anyhow.
Don't continually re-define the method within your call to the then() function. Refactor your code so that it is defined once elsewhere.

edited Nov 13 '13 at 17:13

answered Nov 13 '13 at 01:12

Ben Smith

19,589
6
65
93

tgsagc.continue is set in the getFindDates function when it returns a date which is before the needed dates – SquiresSquire Nov 13 '13 at 02:45
Just been playing around with it, code is virtually the same as in the question, and the then() function in the while loop appears to not be called – SquiresSquire Nov 13 '13 at 09:26
The code in my answer is exactly the same as yours. I am suggesting changes to make. Definetly try removing the return from the function in then(). – Ben Smith Nov 13 '13 at 10:12
the code I was referring to was the one I was editing and testing with 0.o, though, I've noticed with the code currently in the just-edited question the then() function is not being executed? – SquiresSquire Nov 13 '13 at 10:23
1

If you cannot debug this method, then I'd put console log output to work out where your program is getting to before it fails; it could be that you program is failing before it gets to the then() function? – Ben Smith Nov 13 '13 at 10:44
I've tried all sorts of debugging, everything appears fine, but it just stops once it gets to the while loop, without executing the then() function, I even placed a log before and after the then() function, they are logged into the console, but nothing inside the then() function is – SquiresSquire Nov 13 '13 at 11:02
cotinue is also a reserved word in Python. Instead of using continue, try calling it something else e.g.for now change it to "tgsagc.proceed" – Ben Smith Nov 13 '13 at 11:21
let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/41100/discussion-between-squiressquire-and-fresh) – SquiresSquire Nov 13 '13 at 11:22

SquiresSquire · Accepted Answer · 2013-11-25T01:11:15.640

0

We ended up having to scope the function, so it loads the next page in the loop.

This is mainly because CasperJS is not designed to calculate scores, and it tries to asynchronously do the calculation, missing the required functions

edited Nov 25 '13 at 01:11

answered Nov 18 '13 at 01:16

SquiresSquire

2,404
4
23
39

1

You should provide more detail on how you fixed the problem (e.g. code changes, what did you do to "scope the function"?). This answer is not going to help others with a similar problem. – Ben Smith Feb 06 '14 at 15:11

CasperJS Load next page in loop

2 Answers2