CasperJS click() doesn't load new html

Question

I'm trying to use CasperJS' click() to follow a link which generates a modal on the current screen. When I query the proper selector and click it in the browser console using document.querySelector().click() it works, but even when I casper.evaluate() this it doesn't work. I found someone who had a very similar problem, but his question remains unanswered, and I am experiencing almost identical problems. casperjs button click doesn't navigate to next page the code I'm currently using is

this.waitForSelector('div.talk-sharing__tools a.rate-button', function() {
    this.then(function() {
        this.evaluate(function() {
            document.querySelector('a.rate-button').click();
});

the page I'm trying to scrape is http://www.ted.com/talks/uri_alon_why_truly_innovative_science_demands_a_leap_into_the_unknown

The untold truth about the casperjs and phantomjs tags is that nobody can help you solve the problem as long as you don't provide the link to the page in question. The pages you scrape behave very differently. That is why the SO format of posting does not really work for such question. You would have to post the complete page code here, which is really not encouraged. Also, your code should work as is, the page does something funny. We don't want to dream up something that might work. — Artjom B., Jun 13 '14 at 08:56
@ArtjomB. I actually disagree with you on this one. This is a pretty general question, and while scraping questions aren't ideal for SO, there is a concrete usefulness here and I've been able to answer a few of these questions in the past. Also the page is now linked, but I assume that's a more recent development. — Slater Victoroff, Jun 13 '14 at 15:27

Artjom B. · Accepted Answer · 2014-06-13T16:58:30.497

It is really impossible to do those navigation steps with the phantomjs engine. It seems that the QtWebkit fork of phantom (version 1.9.7) is not up to the task anymore. Although your code works fine as is with slimerjs. You can comfortably install slimerjs through npm:

npm install -g slimerjs

and run your script with

casperjs --engine=slimerjs script.js

I tried several things with phantomjs that did not work. I added casper.on listeners for remote.message and resource.error, but those didn't show any errors when running the script in phantom. logLevel: "debug" also didn't show anything.

First: Using a casperjs function.

casper.thenClick("div.talk-sharing__tools a.rate-button");

Second: Trying to explicitly show the modal for the button (specific to the page).

casper.then(function(){
    var name = this.evaluate(function(){
        var modal = document.querySelectorAll(".modal-wrapper > .modal")[0];
        modal.className += " modal--show";
        return modal.className;
    });
    this.echo("name: "+name); // null
});

casper.waitUntilVisible(".modal__head__title");

Third: Let jQuery do the work.

var casper = require('casper').create({
    remoteScripts: [ "http://code.jquery.com/jquery-1.11.1.min.js" ]
});

casper.waitForSelector(selector, function() {
    this.evaluate(function() {
        jQuery("a.rate-button").click();
    });
});

score 0 · Answer 2 · answered Oct 28 '14 at 21:14

0

Encountered similar issue today. After executing this.click() on start page (which I verified that PhantomJS is executing - by hooking into resource.requested and resource.received events), current page remained on start page (and it should go to clicked link page).

Solution was to update to the latest PhantomJS (1.9.8 at this moment).

Please note that I had PhantomJS 1.9.7 when the above described problem occurred.

answered Oct 28 '14 at 21:14

pbradaric

21
3

It probably had some https resources that were necessary for the page to function or the link had the https protocol. See http://stackoverflow.com/a/26417660/1816580 – Artjom B. Oct 28 '14 at 21:43
@ArtjomB. No, in my case there were no https resources that this particular page depended on (nor was the link https). – pbradaric Oct 30 '14 at 00:12
If you look on [this page](http://phantomjs.org/release-1.9.html). There's just no other fix that was introduced that would fix your issue. Maybe there where redirects to https resources. – Artjom B. Oct 30 '14 at 08:09

CasperJS click() doesn't load new html

2 Answers2

Linked