PhantomJS performing a search and grabbing the URL

Question

The aim for my PhantomJS script is to load a specific website, find the search input element, write text in that input and perform the search. From the page loaded containing the search results I just want to grab the whole URL - it will contain the search parameters.

After the interaction with the input, I'm using onUrlChanged() to detect when the search results URL has returned. This never gets called. I don't know whether it is due to the search not being completed or whether the onUrlChanged() is in the wrong place.

var page  = require('webpage').create(),
    response = {};
page.onUrlChanged = function(targetUrl) { // Page loaded
    page.evaluate(function() {
        var inputs = document.getElementsByTagName('input');
        for (var i = 0; i < inputs.length; i++) {
            if (inputs[i].type.toLowerCase().indexOf('search') > -1) {
                inputs[i].value = 'cat dog';
                inputs[i].focus();
                page.sendEvent('keypress', 'Enter');
                break;
            }
        }
    });

    // This does not get called
    page.onUrlChanged = function(targetUrl) { // Search result loaded
        response.content = targetUrl;
        console.log(JSON.stringify(response));
        phantom.exit(1);
    };
};

page.open('http://www.mysearch.com/', function(status) {
    if (status !== 'success') {
        response.content = 'Unable to access network';
        console.log(JSON.stringify(response));
        phantom.exit(1);
    } else {
        phantom.exit(1);
    }
});

Note: I know that CasperJS is better at this kind of use case, however in my current environment I have to use PhantomJS.

Artjom B. · Accepted Answer · 2015-03-11T10:59:32.633

According to the documentation, you need to use page.event.key.Enter not the string 'Enter'.
page.evaluate() is the sandboxed page context. It has no access to variables defined outside and that includes page and phantom. So the page.sendEvent() call must be moved outside.
Then you should move registering the callback before doing the action, otherwise the wrong onUrlChanged callback is called. If an actual new page is loaded, you should use onLoadFinished instead of onUrlChanged.
Don't use a loop, when there are CSS selectors.
onUrlChange is triggered when the URL changes which doesn't mean that the page is already loaded. Use onLoadFinished to wait until the page is loaded (doesn't include dynamic JS). Also, page.open() with a callback is the same as registering the callback to onLoadFinished and invoking page.open() without the callback.

Final complete code:

var page  = require('webpage').create(),
    response = {};
page.open('http://www.mysearch.com/', function(status) {
    if (status !== 'success') {
        response.content = 'Unable to access network';
        console.log(JSON.stringify(response));
        phantom.exit(1);
    }
    var exists = page.evaluate(function() {
        var input = document.querySelector('input[type*="search"]');
        if (input) {
            input.value = 'cat dog';
            input.focus();
        }
        return !!input;
    });
    page.onLoadFinished = function(status) { // Search result loaded
        console.log("new page loaded with status: " + status);
        phantom.exit();
    };
    if (exists) {
        page.sendEvent('keypress', page.event.key.Enter);
    } else {
        console.log("field not found");
    }
};

Thanks, there is a page reload, so I've now used `onLoadFinished` however, it seems nothing in the `evaluate` function is being called. I've tried placing only `page.openUrl('http://google.com')` in there, but it only returns `NULL`. Am I right in placing the `page.open('http://www.mysearch.com/')` after the last block? — Fisu, Mar 11 '15 at 09:59
`page.openUrl` and `page.open` return nothing. You need to use the events like `onLoadFinished` to get the results. I updated my answer which a complete script and 2 more problems. — Artjom B., Mar 11 '15 at 10:27
That now makes more sense. Now `onLoadFinished` is not getting called. Through debugging I can see that `exists` **is** true. Therefore I assume the keypress is not sending properly or the element loses focus before the keypress event. Are one of these possible? Is there another way to submit an input element? On the actual site I can confirm that focusing the element, typing then pressing enter starts a search and loads a new page. — Fisu, Mar 11 '15 at 10:54
You could try to `submit()` the form that surrounds it. Or [click](http://stackoverflow.com/questions/15739263/phantomjs-click-an-element) the search button if there is one. — Artjom B., Mar 11 '15 at 11:04

PhantomJS performing a search and grabbing the URL

1 Answers1