1

We just implemented CI without using headless browser, but actual browser's. So once there is new code checked in by development team.Our smoke automation code runs as background process on auto machine and send out results. It evens send across screenshots if any failures. What we did not get is, what is the advantage of using headless browser to above process. We need our smoke scripts to get this coverage. Say submitting a massive form which can run for within 5 minutes with controls like Select, Calendar, Uploaded, etc and not simple load of web page.

alecxe
  • 462,703
  • 120
  • 1,088
  • 1,195
ReuseAutomator
  • 410
  • 3
  • 14

5 Answers5

10

"Headless browser" is a ambiguous term, in general. People use it to refer to browsers designed to run headlessly but also to refer to a browser that normally requires a real display but is run in a configuration that allows it to run with a virtual display. An instance of the latter would be running Firefox with Xvfb. (The angle I'm taking is descriptive: this is how people talk about browsers that are headless. I'm not talking about whether this is correct or not, which would be a prescriptive angle.)

I don't generally run tests with those browsers designed to run headlessly. Why?

  1. Whenever I compared performance between a headless browser and Chrome, the performance gain was very modest, and this was done naively, without trying to optimize Chrome's setup for speed. (I could imagine having the user data be in ram (/dev/shm) for instance, which I did not try.)

  2. At the end of the day I must certify that my software runs in real browsers used by real people. Normal people don't use browsers designed to be headless to run my software. There are no shortcuts to checking that a web application will work with a specific browser. You have to use the target browser, end of story. Even running the code in browser X version Y is not a guarantee that the code will work in browser X version Z (where Z is different from Y). I've had an application that worked fine in Chrome X-1 and Chrome X+1 but crashed badly in Chrome X (where X is some version number I cannot remember right now). Nothing else than testing in Chrome X revealed this. Whatever a headless browser is based on, it is not a substitute for testing in the actual browser your application is targeting.

  3. Conversely, the headless browser can have quirks of its own that do not show up in other browsers. I recently had to figure out why PhantomJS was behaving differently from Firefox or Chrome when it came to dealing with font subtleties. The time spent figuring out quirks that users won't encounter because they don't use headless browsers could be spent figuring out quirks that users will encounter when they use the browsers that people normally use.

I'm not saying there's no good use for browsers designed to run headlessly, but for testing, I pass.

I do run "headful" browsers headlessly when I run local tests. I don't want the test window to show up on my desktop unless I really need it.

Louis
  • 146,715
  • 28
  • 274
  • 320
  • There is no need for other answers, if there is one like this. Awesome explanation. – alecxe Sep 01 '15 at 17:18
  • Thanks! In all fairness though I imagine someone who has used headless browsers for testing for a substantial period of time and *then* switched away from them would have more to say than I do. :) When it came to using them for testing applications, I performed a few tests that convinced me quickly it was not advantageous. The other cases where I've used them were not testing-related. – Louis Sep 01 '15 at 17:31
  • Headful browsers locally-Thumbs up for that and great explanation. – ReuseAutomator Sep 02 '15 at 07:41
2

headless browsers are relatively faster as it does not require any GUI or plugin to be loaded. Whole execution, i.e. loading of browser to closing of browser happens on memory. It does not require any executable to be worked. It can be used mainly for functional testing where you do not have to test on particular browser and make sure that functionality works fine. Mainly used for smoke testing.

Mayur Shah
  • 518
  • 3
  • 8
  • -Does it give screenshot in case of failures? Also, we do run Smoke on automation machines. Currently, they run as background process similar to if a windows scheduled tasks . It does provide a valuable screenshot of failures, also our smoke is pretty robust & covers major controls mentioned above. How about Angular page automation, does it support it? – ReuseAutomator Sep 02 '15 at 07:23
  • Yes, it gives screenshot in case of failures, we have to handle it via try catch block. I believe it should support Angular pages as well. It needs DOM to be interacted with. Hope I answered your questions. – Mayur Shah Sep 02 '15 at 10:08
  • -Does it handle complex controls (submission of form, validate confirm message in pop ups, or just Navigation, validate page loads, file uploads) – ReuseAutomator Sep 02 '15 at 23:33
0

Headless browsers are usually used to workaround the absence of a real display on machines with no display hardware and no physical input devices.

If you can fire up a real browser on a real display, stick to it.

Though, one of the things that, for instance, Xvfb can make easier - is:

it can be used to test whether clients work correctly at depths or screen sizes that are rarely supported by hardware.

alecxe
  • 462,703
  • 120
  • 1,088
  • 1,195
0

I've still never found headless browsing to be consistent, nor valuable to what I need to get accomplished. You can accomplish anything you need to just fine using a real browser.

As far as advantages go, the only advantage that headless browsing gives, is the ability to test on machines that don't have a display.

ddavison
  • 28,221
  • 15
  • 85
  • 110
0

Some will say that using headless tests will be quicker but the facts state that the difference isn't that big. Also, for debugging purposes it is way easier to use a normal browser.

Homewrecker
  • 1,076
  • 1
  • 15
  • 38