Scrape text inside dynamic element with no id, class, attributes, etc

Question

The only thing I've got going for me is that the preceding <td> will always have the same (and unique to the document) contents:

<td>  
    <label>unique text<label>  
</td>  
<td>dynamic text</td>

I can easily grab it with jQuery in the browser console (the page has jQuery loaded):

$("label:contains('unique text')").parent().next().text();

I've been at this for a while and have tried everything I can think of.

My most recent attempt was to use casperjs' evaluate and:

casper.thenEvaluate(function addID() {  
    $("label:contains('unique text')").parent().next().attr('id', 'uniqueID');  
});  
casper.then(function getText() {  
    var target = this.getHTML('td#uniqueID');  
    this.echo(target);  
});

Which gives me:

CasperError: No element matching selector found: td#uniqueID

Why is my casper.thenEvaluate function not creating the td#uniqueID that I'm looking for?

If I do it like this post's answer:

casper.then(function getText() {  
    this.evaluate(function addID() {  
        $("label:contains('unique text')").parent().next().attr('id', 'uniqueID');  
    });
    var target = this.thenEvaluate(function returnText() {  
        return $('#uniqueID').text();
    });
    this.echo(target);
});

I get an [Object Casper] which seems to be exactly what it sounds like. It's filled with waitForContent,scrollTo, etc...

note: The above code block was incorrect (as was pointed out in this answer by Artjom B.) and was changed to this:

casper.then(function getText() {  
    this.evaluate(function addID() {  
        $("label:contains('unique text')").parent().next().attr('id', 'uniqueID');  
    });
    var target = this.fetchText('#uniqueID');
    this.echo(target);
});

The problem still persisted. See my answer below for the resolution.

Artjom B. · Answer 1 · 2015-05-07T11:43:55.867

2

If you already tried it like in the linked answer, why not just copy it? Your error is that you use thenEvaluate inside the then block. CasperJS works in steps and you scheduled a step where it is not necessary. This creates another step which is executed later.

Change thenEvaluate to evaluate and it should work fine. While you're at it, you could combine the two:

casper.then(function getText() {  
    var target = this.evaluate(function addID() {  
        $("label:contains('unique text')").parent().next().attr('id', 'uniqueID');  
        return $('#uniqueID').text();
    });
    this.echo(target);
});

or even

casper.then(function getText() {  
    this.evaluate(function addID() {  
        $("label:contains('unique text')").parent().next().attr('id', 'uniqueID');  
    });
    var target = this.fetchText(#uniqueID);
    this.echo(target);
});

edited May 07 '15 at 11:43

answered May 12 '14 at 07:08

Artjom B.

61,146
24
125
222

Thank you very much for this, I had actually tried it about 99 different ways (including something like this^), but I had no idea which way was the right way, I only knew that I ended up about with about the same result regardless. I will update my original post to reflect the correct way to call these. – Seth Brasile May 13 '14 at 18:14
You should **not** edit your answer to replace the faulty code with working code, because the next user that has the same problem might not understand what was the problem and how it was solved. – Artjom B. May 13 '14 at 19:14
I'm sorry, it was more a typo than the actual problem so I thought it best to clear up the code. I'll edit it to include both. – Seth Brasile May 13 '14 at 21:01

Seth Brasile · Accepted Answer · 2014-05-13T19:10:53.613

I finally got it resolved. When I visit the page in a browser, I click a link and javascript calls a modal via ajax POST with the information I'm grabbing. I assumed that in casperjs, it would work the same way when I use:

casper.thenOpen('http://www.website.net/details.php', {
    method: 'post',
    data:   {
        'id': 'foo',
        'key':  'bar',
    }
});

It looks like casper renders this slightly differently than I had anticipated. Even though the main page had jQuery loaded, the modal did not. So I needed:

clientScripts: ["jquery-2.1.1.min.js"]

Inside my:

var casper = require('casper').create({

});

(casperjs docs on jQuery)

Here is the full script for reference:

var url = "http://www.website.net/search.php?key=foobarfoobarfoobarfoobar";

var casper = require('casper').create({
    clientScripts: ["jquery-2.1.1.min.js"]
});

casper.start(url, function() {
    //some other unrelated stuff is going to go here
});

casper.thenOpen('http://www.website.net/details.php', {
    method: 'post',
    data:   {
        'id': 'foo',
        'key':  'bar',
    }
});

casper.then(function getText() {  
    this.evaluate(function addID() {  
        $("label:contains('unique text')").parent().next().attr('id', 'uniqueID');  
    });
    var target = this.fetchText('#uniqueID');
    this.echo(target);
});  

casper.run();

Scrape text inside dynamic element with no id, class, attributes, etc

2 Answers2