152

We are using Selenium to automate our UI testing. Recently we have seen majority of our users using Chrome. So we wanted to know - pros and cons of using PhantomJS vs Selenium:

  • Is there any real advantage in terms of performance, e.g. time taken to execute the test cases?
  • When should one prefer PhantomJS over Selenium?
Hash
  • 4,647
  • 5
  • 21
  • 39
spirit3189
  • 1,565
  • 3
  • 11
  • 9

5 Answers5

184

They are attacking different problems. Since PhantomJS runs perfectly on the command-line, it is suitable as the first layer of smoke testing, whether as part of development workflow and/or in a continuous integration server. Selenium targets multiple browsers and hence it is very useful to ensure cross-browser consistency and carry out extensive testings across different operating systems.

If your web application needs to run on a variety of web browsers, running the UI testing only with PhantomJS will not yield the most test coverage. However, it is perfectly fine to launch PhantomJS and exercise some basic sanity checks before doing the in-depth tests. Imagine the madness of testing a finance application where the login screen is unintentionally broken and non-functional!

Note that the line between the two gets slightly blurred with the recent WebDriver support in the latest PhantomJS. It is now possible to quickly run the tests first using PhantomJS and then (assuming there is no serious error encountered) continue to execute the same tests thoroughly in a Selenium setup.

Ariya Hidayat
  • 12,523
  • 3
  • 46
  • 39
  • Thanks for the answer.Any link that can help me understand how people are using phantomjs to collect performance related stats in a real world production app. – spirit3189 Dec 31 '12 at 13:11
  • 35
    Ariya Hidayat looks to be the creator of PhantomJs – Sebastian Patten Oct 14 '14 at 16:23
  • It seems that your described workflow of starting to write tests with PhantomJS then use Selenium assumes a total rewrite of tests... Could you use PhantomJS scripts directly with Selenium? – lajarre Aug 20 '15 at 09:45
50

With the recent WebDriver integration (as Ariya has noted), you can now use Selenium to drive PhantomJS.

This is immensely powerful.

You can run a set of fully automated Selenium tests (using PhantomJS as the WebDriver implementation) via your CI on a headless Unix server on every check-in. Then if you want to test browser compatibility you can run your tests locally by changing the underlying WebDriver implementation to Chrome, Firefox etc.

mekondelta
  • 993
  • 7
  • 17
41

I am currently writing a web extraction framework. I have 524 tests that get data from 250 websites using XPath. Initially the framework used a HTML parser, HTMLCleaner, but I am currently investigating using Selenium because I want Javascript support. I have run the tests against the HtmlUnit, Chrome, Firefox and PhantomJS drivers. Here is a comparison of the time taken and the number of failures for each approach:

                    Failures    Time (secs) 
HtmlCleaner         0           82  
HtmlUnit            169         102 
Google Chrome       38          562 
Firefox             46          1159    
PhantomJS           40          575

Some comments:

  • In some cases the "failures" may not be failures at all, it may be that the extractors are failing because Javascript is re-writing the DOM. I am in the process of analyzing the failures to find the cause.

  • That said, HtmlUnit is the fastest Selenium driver but it is also unreliable. This unreliability does not just concern Javascript, there are problems processing "messy, dirty, real-world" HTML because something seems to be broken in the tag balancing algorithm. A couple of issues have been raised about this but they have not been fixed - see HTML-UNIT 1423 and HTML-UNIT 1046.

  • Firefox is the slowest Selenium driver, even though I am disabling image loading and stylesheets. This is because it is the slowest to load and initialize, making it considerably slower than Chrome, and every time an extraction fails I need to reload the driver (in the tests I create a pool of 5 drivers to mitigate the URL retrieval delays for all the Selenium web drivers).

  • PhantomJS achieves a better accuracy than Firefox, slightly lower than Chrome, but in around half the time of Firefox. What is more, I can run it on my dev box, it does not "take over my machine" by launching multiple browsers so I can get on with work.

I would highly recommend PhantomJS.

Mark Butler
  • 4,361
  • 2
  • 39
  • 39
  • 1
    It takes 9 minutes to run your test suite with phantomJS? That must feel like forever... – Kevin Nov 06 '13 at 07:46
  • @Kevin Yes :) - but HTMLCleaner is the standard test, I mark use [JUnit categories](http://stackoverflow.com/questions/15418466/how-to-include-multiple-categories-in-junit4) to mark the other tests as optional so they aren't part of the standard unit tests – Mark Butler Nov 07 '13 at 06:37
  • Thanks. Yeah I've been working on full-js tests and they're so glacial - like 15-20 seconds for a few page test. Perhaps my scale of 'glacial' needs to be adjusted though hehe :) Odd though that if I do it manually, it only takes ~5 seconds to click through the forms. – Kevin Nov 08 '13 at 19:01
  • Hey Mark, it's been two years since this answer. What did you ended up choosing? – lucaswxp Oct 10 '15 at 11:56
  • 1
    @lucaswxp Yes! And lots of other projects in between. As I explain above there is no perfect choice. At the time, I used HtmlCleaner, but added an option to use PhantomJS, if the page to be extracted needed it. – Mark Butler Oct 11 '15 at 18:41
  • @MarkButler: is your web extraction platform open sourced? – iconoclast Jul 18 '16 at 18:13
  • 1
    @iconoclast No - unfortunately - previous employer has the IP. – Mark Butler Jul 18 '16 at 19:01
2

Leveraging the Power of Both Selenium and PhantomJS PhantomJS has the headless browser capabilities, hence it is good to use it as one of the browsers with selenium (In addition to traditional browsers like IE, Chrome etc.,) Advantages of this approach:

  1. Can be used for doing Sanity for web applications in CI (Even though the agent machines doesn't have IE or chrome) the tests will run effectively.
  2. Some development teams use this approach to get quick results and consumes less amount of time and resource.
  3. The greatest feature of phantom JS is the screen capture, running parallel tests using multi threading, which will reduce your execution time tremendously.
Muthu ragavendra
  • 31
  • 2
2

challenges I faced while using PhantomJS:

My Application was pricing web application:

  1. At some point locators that were working fine on chrome browser are not working on phantomJS.
  2. Sometimes while performing double click or context click by selenium I have to put an extra check for verifying the operations coz at first place it is not clicking.
  3. Cache and cookies persistency. After doing log out and then log in the data remain in cache. | So we perform the test on chrome.
  4. One of the most important issue I found "File uploading issue". We can not upload a file in phantomJs browser in our application. We tried many things javascriptexcutor, jqueries but none of them worked at all. So we also perform this test on Chrome. Note: We used javascript functions extensively in our framework to interact with web element for PhantomJS. One thing is sure the execution time is very less in PhantomJS. So it depends upon client priority whether he wants Functionality/Performance then go with it. If you want to test end to end scenario then go for chrome.
Vaibhav_Sharma
  • 546
  • 1
  • 9
  • 22