Zentest - How to stop automatic testing when red

Question

I am trying to configure autotest so that when I run my test suite and I have a failing test, autotest stops testing and waits for me to make a change before testing again. With my current configuration autotest keeps testing indefinetly when it encounters a failing test making it a bit of a hassle to deal with (having to tab into terminal and stop the autotest server everytime I get a failing test).

I am working on a rails app using RSpec, Zentest and Spork.

Relevant Gem Versions:

autotest (4.4.6)
rspec (2.6.0)
rspec-core (2.6.4)
rspec-expectations (2.6.0)
rspec-mocks (2.6.0)
rspec-rails (2.6.1)
spork (0.9.0.rc8)
ZenTest (4.5.0)

My .autotest file:

module Autotest::Notify
  def self.notify title, msg, img, pri='low', time=3000
    `notify-send -i #{img} -u #{pri} -t #{time} '#{msg}'`
  end

  Autotest.add_hook :ran_command do |autotest|
    results = [autotest.results].flatten.join("\n")
    output = results.slice(/(\d+)\s+examples?,\s*(\d+)\s+failures?(,\s*(\d+)\s+pending)?/)
    folder = "~/.autotest_icons/"
    if output  =~ /[1-9]\d*\sfailures?/
      notify "FAIL:", "#{output}", folder+"failed.png", 'critical', 10000
    elsif output  =~ /[1-9]\d*\spending?/
      notify "PENDING:", "#{output}", folder+"pending.png", 'normal', 10000
    else
      notify "PASS:", "#{output}", folder+"passed.png"
    end
  end
end

Note: Most of my .autotest file was to get popups working in linux to display if my tests are passing or failing.

I have been searching for an answer to this problem for a while and have had no luck and I have found it very difficult to get my hands on some good documentation for Autotest. I have been staring at the RDoc for Zentest for quite a while now as well and I must just be missing something.

Any help, links to examples, etc, would be greatly appreciated.

I am wondering if your `.autotest` file is doing something to make it autotest recurse. I would try to remove possible causes one at a time. For example, remove `spork` from the mix, run autotest with a known error and see if it keeps going. If it does, move your `.autotest` file somewhere temporarily, effectively running without an `.autotest` file and try again. Whenever I run into a non-obvious problem I try to walk the logic chain and make small changes until I can isolate the problem. — Midwire, Jul 11 '11 at 15:55
PS: Autotest should never keep running tests after a given code change. It should run the affected tests, report the results (green or red) and then stop and wait for more changes. Something in your configuration is causing this anomaly. — Midwire, Jul 11 '11 at 15:58
Thanks for the response. To be honest the problem seems to be rather random. I have been running with the same configuration now for a while and autotest will only re-run tests when they are red once in a while (maybe 1/10 times). I can't really find a relation between the tests I am running and the strange autotest behavior but I will keep trying. Thanks for your advice and if I end up getting a definitive answer then I will post it. My main problem now is that every time I think I have fixed it, it ends up behaving the same just less often. — Chris Knadler, Jul 11 '11 at 16:30
Interesting. I'd like to see the answer if you find one. Good luck. — Midwire, Jul 11 '11 at 19:09
Try running with "autotest -v", which will show you what file change triggered the rerun. Then add an exception to .autotest. — Jonathan del Strother, Jul 13 '11 at 11:20

score 4 · Accepted Answer · answered Sep 24 '11 at 22:36

I had the same problem and found that my development server was writing to the log file. After I added this to my .autotest file, the problem went away:

Autotest.add_hook :initialize do |at|
  at.add_exception(%r{^\./\.git})
  at.add_exception(%r{^\./log})
end

score 1 · Answer 2 · edited May 23 '17 at 10:24

I saw a similar problem with ZenTest when I had a gem that was writing data to a directory that ZenTest was monitoring. IIRC, it was a gem that did full-text searching -- the index file the search generated was triggering ZenTest to run again, thereby regenerating the index.

I tracked down the problem by modifying the Growl notifications to tell me which files were triggering the autotest run (I was running on a Mac at the time).

The solution was to add an exception/exclude to the .autotest file, to tell it to ignore the index.

(I've just seen : Spork is repeatedly re-running failing tests in autotest which sounds very similar to your problem)

Zentest - How to stop automatic testing when red

2 Answers2