15

I have been having problems with my tests timing out and failing randomly. I have been looking around for best practices of how to write robust capybara integration tests but I don't find anything helpful.

Ever since we started writing a bunch of tests our CI server has been failing randomly, making our app look ver unstable, but the tests always (mostly) pass locally.

I want to find out how you and other experienced BDD/TDD gurus handle:

  • How to deal with external javascript and stuff (KissMetrics, Google Analyics, etc) and
  • Debugging and preventing timeout errors that break the build

Any help would be appreciated.

PeppyHeppy
  • 1,345
  • 12
  • 20
  • Which versions of poltergeist and phantomjs are you using? I've encountered seemingly random timing issues with poltergeist > 0.7.0 and phantomjs 1.7.0. They went away when we fixed poltergeist at 0.7.0 and phantomjs < 1.7.0 (I forget which exact number but I think 1.7.0 is when the trouble started). – Josh Rieken Jan 12 '13 at 16:55
  • I am using poltergeist (1.0.2) and phantomjs 1.7.0. I just noticed that [phantomjs 1.8.9 was released](http://phantomjs.org/release-1.8.html), but I haven't tried that. I will upgrade and test it out. With all of the trouble we have been having I wondered if anyone was using these tools. :) – PeppyHeppy Jan 12 '13 at 20:16
  • Cool. Please report back here with your findings. Curious to know whether my problems might be solved, too. :-) – Josh Rieken Jan 12 '13 at 20:17
  • I upgraded to phantomjs 1.7.0 to 1.8.1 and nothing changes. I have the latest poltergeist. (I still get timeouts and element not found errors masked as timeouts) – PeppyHeppy Jan 12 '13 at 21:23
  • Nearly 4 years later. This still seems to be a common problem. While these two answers are some of the problem, there still seem to be many other things that can cause random failures. – phylae Dec 02 '16 at 03:25

2 Answers2

7

To follow up on this. I appreciated @jonleighton's invitation to file a bug on poltergeist, but the problems that I encountered were related to two separate problems:

  1. Bad/wrong capybara assertions that would timeout because I was not following the suggestions clearly outlined in this post
  2. 3rd party javascripts and things. Basically I had 3rd party javascripts like kissmetrics, google analytics, and even live help chat that would load each time a test was wrong, I eliminated this from happening and my tests got faster and appear to be more stable/consistent.
PeppyHeppy
  • 1,345
  • 12
  • 20
  • This doesn't explain exactly what was the cause of it. Something in the 3rd party scripts was causing it, do you have any clue? – joaomilho Jan 22 '14 at 10:53
  • @joaomilho the problems that I was encountering were related to timeouts and bad assertions in the test, specifically replacing `page.should_not have_content('blah')` with `page.should have_no_content('blah')` – PeppyHeppy Feb 18 '14 at 21:03
3

I have found that in many cases I can help my integration tests be more deterministic by taking advantage of Poltergeist's blacklist feature. In my case, I have blacklisted host names like the following.

typekit.net
facebook.net
facebook.com
google.com
google-analytics.com
...

The idea is to turn off anything that a) is not really needed for testing and b) could affect the page load completion under the CI environment. I have noticed this helps tremendously. Also, using this PhantomJS option with Poltergeist helps too:

phantomjs_options: ['--ignore-ssl-errors=yes']
MetaSkills
  • 1,954
  • 18
  • 15
  • Phantomjs docs specify using true/false for --ignore-ssl-errors, although yes/no is also supported. – SamH Mar 18 '17 at 23:16