11

I'm barely started to use Jenkins and this is the first problem I've had so far. Basically my jenkins job always succeed even when an error happened in some of the tests. This is what I'm running in the shell config:

bundle install 
rake db:migrate:reset
rake test:units
rake spec:models

Thing is that Jenkins only reports a failure when the task which fails is the last one. For instance, if I put "rake test:units" the last task it will notify an error if something go wrong. Using this configuration I only get error reports for the rspec tests but not for the unit tests.

Anyone wondering why I don't only use rspec or unit test, we are currently migrating to rspec but this problem is still painful.

This is part of the log from Jenkinsm as you can see one of the unit test fails but jenkins still finish with success.

314 tests, 1781 assertions, 1 failures, 0 errors, 0 skips
rake aborted!
Command failed with status (1): [/var/lib/jenkins/.rvm/rubies/ruby-1.9.3-p1...]

Tasks: TOP => test:units
(See full trace by running task with --trace)
Lot of rspec tests here....
Finished in 3.84 seconds
88 examples, 0 failures, 42 pending
Pushing HEAD to branch master of origin repository
Pushing HEAD to branch master at repo origin
Finished: SUCCESS
Marcos Holgado Molina
  • 416
  • 3
  • 10

3 Answers3

18

Jenkins executes the commands you type into a Build Step box by writing them to a temporary file and then running the script using /bin/sh -xe.

Usually this produces the desired effect: Commands are executed in sequence (and printed) and the script aborts immediately when a command fails i.e. exits with non-zero exit code.

If this is not happening to you, the only reason can be that you have overridden this behavior. You can override it by starting the first line of your Build Step with these two characters: #!.

For example, if your Build Step looks like this:

#!/bin/bash
bundle install
rake db:migrate:reset
rake test:units
rake spec:models

Then it means Jenkins will write the script to a temporary file and it will be executed with /bin/bash. When invoked like that, bash will execute commands one-by-one and not care if they succeed. The exit code of the bash process will be the exit code of the last command in the script and that will be seen by Jenkins when the script ends.

So, take care in what you put on the first line of the Build Step. If you do not know how shell works, do not put a hash-bang at all and let Jenkins decide how the script should be run.

If you need more control over how the Build Step is executed, you should study the man page of the shell you use to find out how to make it behave the way you want. Jenkins doesn't have much of a role in here. It just executes the shell you wanted the way you wanted.

sti
  • 11,047
  • 1
  • 27
  • 27
  • 1
    You were right indeed, I've had "#!/bin/bash" as the first line :) – Marcos Holgado Molina Sep 11 '12 at 06:24
  • 1
    if you add `#!/bin/bash -x` you can get the part where Jenkins shows the command it is running in the script output. Really it's just the `-e` that you don't want from Jenkins' default `-xe` – ZombieDev Sep 13 '13 at 23:00
5

Jenkins can only see the result code of the last command run so it has no way of knowing what the result of rake test:units is.

The easiest thing is probably to have each command of those commands as a separate jenkins build step.

Frederick Cheung
  • 83,189
  • 8
  • 152
  • 174
  • This answer is 100% correct, but I do not think it is a good idea to make people think every command as a separate build step is best practice in Jenkins. See my answer below. – sti Jul 13 '12 at 21:01
  • Having a sub-step for each breakable part of the job is not such a bad idea, as long as it is clear that each sub-step is run independantly. Another option is to check the result of each sub-step in the build and exit with error, if needed. – Gonen Jul 14 '12 at 00:35
  • One downside to this solution is that if you are publishing test results (such as with the rspec JUnitFormatter + Jenkins JUnit publisher), having separate script steps that each can fail the build will make your test trend graph have weird up/down jumps as some tests suites don't even run. I think @sti has a better solution in his answer. – ZombieDev Sep 13 '13 at 23:04
5

An alternative solution is change your first line to the following:

#!/bin/bash -e

This tells your script to fail if any of the commands in the script return an error.

See: Automatic exit from bash shell script on error

Community
  • 1
  • 1
Dave Brace
  • 1,809
  • 2
  • 16
  • 20