Taking reliable screenshots of websites? Phantomjs and Casperjs both return empty screen shots on some websites

Question

Open a web page and take a screenshot.

Using ONLY phantomjs: (this is a simple script, in fact it is the example script used in their docs. http://phantomjs.org/screen-capture.html

var page = require('webpage').create();
page.open('http://github.com/', function() {
  page.render('github.png');
  phantom.exit();
});

Problem is that for some websites (like github) funny enough are somehow detecting and not serving phantomjs and nothing is being rendered. Result is github.png is a blank white png file.

Replace github with say: "google.com" and you get a nice (proper) screenshot as is intended.

At first I thought this was a Phantomjs issue so I tried running it through Casperjs with:

casper.start('http://www.github.com/', function() {
    this.captureSelector('github.png', 'body');
});

casper.run();

But I get same behavior as with Phantomjs.

So I figured ok this is most likely a user agent issue. As in: Github sniffs out Phantomjs and decides not to show the page. So I set the user agent like below but that still didn't work.

var page = require('webpage').create();
page.settings.userAgent = 'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2049.0 Safari/537.36';
page.open('http://github.com/', function() {
  page.render('github.png');
  phantom.exit();
});

So then I tried to parse the page and apparently some sites (again like github) don't appear to be sending anything down the wire.

Using casperjs I tried to print the title. And for google.com I got back Google but for github.com I got back bupkis. Example code:

var casper = require('casper').create();

casper.start('http://github.com/', function() {
    this.echo(this.getTitle());
});

casper.run();  

The same as above also produces the same result in purely phantomjs.

Update:

Could this be a timing issue? Is github just super slow? I doubt it but lets test anyway..

var page = require('webpage').create();
page.open('http://github.com', function (status) {
    /* irrelevant */
   window.setTimeout(function () {
            page.render('github.png');
            phantom.exit();
        }, 3000);
});

And the result is still bupkis. So no it's not a timing issue.

1. How are some sites like github blocking phantomjs?
2. How can we reliably take screenshots of ALL webpages? Required to be fast, and headless.

Answer 1

After bouncing this around for some time I was able to narrow down the problem. Apparently PhantomJS uses a default ssl of sslv3 which causes github to refuse the connection due to a bad ssl handshake

phantomjs --debug=true github.js

Shows output of:

. . .
2014-10-22T19:48:31 [DEBUG] WebPage - updateLoadingProgress: 10 
2014-10-22T19:48:32 [DEBUG] Network - Resource request error: 6 ( "SSL handshake failed" ) URL: "https://github.com/" 
2014-10-22T19:48:32 [DEBUG] WebPage - updateLoadingProgress: 100 

So from this we can conclude that no screen was taken because github was refusing the connection. Great that makes perfect sense. So let's set SSL flag to --ssl-protocol=any and lets also ignore ssl-errors with --ignore-ssl-errors=true

phantomjs --ignore-ssl-errors=true --ssl-protocol=any --debug=true github.js

Great success! A screenshot is now being rendered and saved properly but debugger is showing us a TypeError:

TypeError: 'undefined' is not a function (evaluating 'Array.prototype.forEach.call.bind(Array.prototype.forEach)')

  https://assets-cdn.github.com/assets/frameworks-dabc650f8a51dffd1d4376a3522cbda5536e4807e01d2a86ff7e60d8d6ee3029.js:29
  https://assets-cdn.github.com/assets/frameworks-dabc650f8a51dffd1d4376a3522cbda5536e4807e01d2a86ff7e60d8d6ee3029.js:29
2014-10-22T19:52:32 [DEBUG] WebPage - updateLoadingProgress: 72 
2014-10-22T19:52:32 [DEBUG] WebPage - updateLoadingProgress: 88 
ReferenceError: Can't find variable: $

  https://assets-cdn.github.com/assets/github-fa2f009761e3bc4750ed00845b9717b09646361cbbc3fa473ad64de9ca6ccf5b.js:1
  https://assets-cdn.github.com/assets/github-fa2f009761e3bc4750ed00845b9717b09646361cbbc3fa473ad64de9ca6ccf5b.js:1

I checked the github homepage manually just to see if a TypeError existed and it does NOT.

My next guess is that the assets aren't loading quick enough.. Phantomjs is faster than a speeding bullet!

So lets try to slow it down artificially and see if we can get rid of that TypeError...

var page = require('webpage').create();
page.settings.userAgent = 'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2049.0 Safari/537.36';
page.open('http://github.com', function (status) {
   window.setTimeout(function () {
            page.render('github.png');
            phantom.exit();
        }, 3000);
});

That didn't work... After a closer inspection of the image - it is clear that some elements are missing. Mainly some icons and the logo.

Success? Partially because we are now at least getting a screen shot where earlier, we weren't getting a thing.

Job done? Not exactly. Need to determine what is causing that TypeError because it preventing some assets from loading and distorting the image.

Additional

Attempted to recreate with CasperJS --debug is very ugly and hard to follow compared to PhantomJS:

casper.start();
casper.userAgent('Mozilla/5.0 (Macintosh; Intel Mac OS X)');
casper.thenOpen('https://www.github.com/', function() {
    this.captureSelector('github.png', 'body');
});

casper.run();

console:

casperjs test --ssl-protocol=any --debug=true github.js

Further the image is missing the same icons but is also visually distorted. Being that CasperJs relies on Phantomjs, I do not see the value in using it for this specific task.

If you would like to add to my answer, please share your findings. Very interested in a flawless PhantomJS solution

Update #1 : Removing the TypeError

@ArtjomB points out that Phantomjs does not support js bind in it's current version as of this update (1.9.7). For this reason he explains: ArtjomB: PhantomJs Bind Issue Answer

The TypeError: 'undefined' is not a function refers to bind, because PhantomJS 1.x doesn't support it. PhantomJS 1.x uses an old fork of QtWebkit which is comparable to Chrome 13 or Safari 5. The newer PhantomJS 2 will use a newer engine which will support bind. For now you need to add a shim inside of the page.onInitialized event handler:

Ok great, so the following code will take care of our TypeError from above. (But not fully functional, see below for details)

var page = require('webpage').create();
page.settings.userAgent = 'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2049.0 Safari/537.36';
page.open('http://github.com', function (status) {
   window.setTimeout(function () {
            page.render('github.png');
            phantom.exit();
        }, 5000);
});
page.onInitialized = function(){
    page.evaluate(function(){
        var isFunction = function(o) {
          return typeof o == 'function';
        };

        var bind,
          slice = [].slice,
          proto = Function.prototype,
          featureMap;

        featureMap = {
          'function-bind': 'bind'
        };

        function has(feature) {
          var prop = featureMap[feature];
          return isFunction(proto[prop]);
        }

        // check for missing features
        if (!has('function-bind')) {
          // adapted from Mozilla Developer Network example at
          // https://developer.mozilla.org/en/JavaScript/Reference/Global_Objects/Function/bind
          bind = function bind(obj) {
            var args = slice.call(arguments, 1),
              self = this,
              nop = function() {
              },
              bound = function() {
                return self.apply(this instanceof nop ? this : (obj || {}), args.concat(slice.call(arguments)));
              };
            nop.prototype = this.prototype || {}; // Firefox cries sometimes if prototype is undefined
            bound.prototype = new nop();
            return bound;
          };
          proto.bind = bind;
        }
    });
}

Now the above code will get us a screenshot same as we were getting before AND debug will not show a TypeError so from the surface, everything appears to work. Progress has been made.

Unfortunately, all of the image icons [logo, etc] are still not loading correctly. We see some sort of 3W icon not sure where thats from.

Thanks for the help @ArtjomB