5

At work we have an automation suite for one of our apps that is set to run on every commit to master/deploy to prod and for a long long time we've been having issues trying to make it reliable enough.

The tests are run in Jenkins using Selenium webDriver + Chrome + xvfb as the display server. We know the tests work because if we run them locally on our laptops (me and 2 friends had tried it) they run perfectly every single time but they fail almost every time in Jenkins. The almost being the keyword as they do work there too... just once every 5 times or so.

And when they fail they don't always fail at the same spot. Sometimes it's a timeout while waiting for an element, sometimes the test ends up in an error page that shouldn't have reached in the first place and I have no idea how it got there... So yeah, it's frustrating.

We've tried a lot of different approaches to debug it. Re-writing the setup of each test to make sure everything is cleared up at the end of every single test so that the next one starts with a clean workspace/cache, making it so Selenium takes screenshots everytime it fails to see what happened, tried different versions of chromedriver/chrome/selenium, put the tests to run several times in a row to see if there was any pattern... and since I'm running out of ideas I've figured I could ask here!.

Is there any way I could set Selenium driver to record the entire flow instead of just taking a screenshot every here and there?. I would LOVE to be able to watch the tests running in Jenkins but I have no idea if that's possible or how to do it.

The tests use Java7 + Spring + TestNG 6.8.1 + Selenium 2.45.0 + XVFB as the display server.

Any thoughts or hints on what I could try?.

Laucien
  • 511
  • 2
  • 5
  • 17
  • First suggestion you can update *almost* all of the frameworks you are using to improve their efficiency. Next suggestion Implicit wait or proper Explicit wait. Final suggestion if it runs good on local, configure jenkins machine **same as your local**. – Naman Jun 07 '16 at 14:02

3 Answers3

6

Reading all the solutions you have tried, I can almost be certain that you use a virtual machine (VM) to execute the Selenium tests. The fact that points to that is mainly:

And when they fail they don't always fail at the same spot. Sometimes it's a timeout while waiting for an element, sometimes the test ends up in an error page that shouldn't have reached in the first place and I have no idea how it got there...

I've been battling the same for more than a two years now and believe me when I say I know

So yeah, it's frustrating.

The main issue with VMs is that they don't have proper rendering and no matter how big their CPU and RAM is, they eventually fail to reproduce your local (or any physical) video card. Things get really ugly when you try to run all your tests in parallel. For example my team couldn't realistically run more than 4 browsers on a VM with 12 GB RAM !?! My laptop with 8 GB easy runs 15 browsers, while I continue to do my daily work. So bottom line: the sync between the Selenium driver and the browser is the issue. It's different every time depending from the rendering speed.

Is there any way I could set Selenium driver to record the entire flow instead of just taking a screenshot every here and there?

Couple of ways actually. But keeping things simple - use a second Thread that will make a sreenshoot on every 5 sec (or as much your Server can keep up with) and keep those in a folder per test. But for performance reasons I would advise to use a Queue and keep just the last 12 (last minute).

UPDATE:

I've extended all this in a flaky tests blog post.

ekostadinov
  • 6,880
  • 3
  • 29
  • 47
  • 1
    You know, I think that was actually it. We hadn't even considered how a VM would render/use the resources differently than an actual physical machine. We're researching that now and also adding some lines so that each test will automatically run again before assuming it failed. – Laucien Jun 11 '16 at 04:36
1

Jenkins is notorious and task scheduler is inefficient.

So use both!

  1. Create a Bat file where you call the selenium scripts

  2. If you use maven as build tool then write this in a notepad and save it with .bat extension

    CD C:\ProjectPath
mvn test
  3. Create a task scheduler job to run a bat file.

  4. Call that "task scheduler job" from Jenkins by:

    • Adding this build step "Run windows batch command"

          schtasks /Run /TN "Task Name"
  5. Congrats, now your test's will run in interactive mode.
S Krishna
  • 1,255
  • 13
  • 9
0

This is Jenkins slaves ( running as service) priority issue.

You need to change Jenkins slaves priority to high or realtime Find Jenkins slave process and increase priority. follow - https://www.sevenforums.com/tutorials/83361-priority-level-set-applications-processes.html

Abhinav Kumar
  • 1
  • 1