We have a system that has some Bash scripts running besides Java code. Since we are trying to test everything that could possibly break, and those Bash scripts may break, we want to test them.
The problem is it is hard to test Bash scripts.
Is there a way or a best practice to test Bash scripts? Or should we quit using Bash scripts and look for alternative solutions that are testable?
There is actually a shunit2, an xUnit based unit test framework for Bourne based shell scripts. I haven't used it myself, but it might be worth checking out.
Similar questions have been asked before:
TAP-compliant Bash testing: Bash Automated Testing System
TAP, the Test Anything Protocol, is a simple text-based interface between testing modules in a test harness. TAP started life as part of the test harness for Perl but now has implementations in C, C++, Python, PHP, Perl, Java, JavaScript, and others.
I got the following answer from a discussion group:
it's possible to import (include, whatever) a procedure (function, whatever it's named) from an external file. That's the key to writing a testing script: you break up your script into independent procedures that can then be imported into both your running script and your testing script, and then you have your running script be as simple as possible.
This method is like dependency injection for scripts and sounds reasonable. Avoiding Bash scripts and using more testable and less obscure language is preferable.
I created shellspec, because I wanted a easy-to-use and useful tool.
It written by a pure POSIX shell script. It has been tested with many shells more than shunit2. It has more powerful features than bats/bats-core.
For example, it supports nested block, easy to mock/stub, easy to skip/pending, parameterized tests, assertion line number, execute by line number, parallel execution, random execution, TAP/JUnit formatter, coverage and CI integration, profiler, etc.
See the demo on the project page.
Nikita Sobolev wrote an excellent blog post comparing a few different Bash test frameworks, Testing Bash applications. These are the frameworks listed in that post:
For the impatient: Nikita's conclusion was to use Bats, but it appears that Nikita missed the Bats-core project which appear to me to be the one to use going forward as the original Bats project has not been actively maintained since 2013.
I can't believe no one talked about OSHT! It's compatible with both TAP and JUnit, it's pure shell (that is, no other languages involved), it works standalone too, and it's simple and direct.
Testing looks like this (snippets taken from the project page):
#!/bin/bash
. osht.sh
# Optionally, indicate number of tests to safeguard against abnormal exits
PLAN 13
# Comparing stuff
IS $(whoami) != root
var="foobar"
IS "$var" =~ foo
ISNT "$var" == foo
# test(1)-based tests
OK -f /etc/passwd
NOK -w /etc/passwd
# Running stuff
# Check exit code
RUNS true
NRUNS false
# Check stdio/stdout/stderr
RUNS echo -e 'foo\nbar\nbaz'
GREP bar
OGREP bar
NEGREP . # verify empty
# diff output
DIFF <<EOF
foo
bar
baz
EOF
# TODO and SKIP
TODO RUNS false
SKIP test $(uname -s) == Darwin
A simple run:
$ bash test.sh
1..13
ok 1 - IS $(whoami) != root
ok 2 - IS "$var" =~ foo
ok 3 - ISNT "$var" == foo
ok 4 - OK -f /etc/passwd
ok 5 - NOK -w /etc/passwd
ok 6 - RUNS true
ok 7 - NRUNS false
ok 8 - RUNS echo -e 'foo\nbar\nbaz'
ok 9 - GREP bar
ok 10 - OGREP bar
ok 11 - NEGREP . # verify empty
ok 12 - DIFF <<EOF
not ok 13 - TODO RUNS false # TODO Test Know to fail
The last test shows as "not ok", but the exit code is 0 because it's a TODO. One can set verbose as well:
$ OSHT_VERBOSE=1 bash test.sh # Or -v
1..13
# dcsobral \!= root
ok 1 - IS $(whoami) != root
# foobar =\~ foo
ok 2 - IS "$var" =~ foo
# \! foobar == foo
ok 3 - ISNT "$var" == foo
# test -f /etc/passwd
ok 4 - OK -f /etc/passwd
# test \! -w /etc/passwd
ok 5 - NOK -w /etc/passwd
# RUNNING: true
# STATUS: 0
# STDIO <<EOM
# EOM
ok 6 - RUNS true
# RUNNING: false
# STATUS: 1
# STDIO <<EOM
# EOM
ok 7 - NRUNS false
# RUNNING: echo -e foo\\nbar\\nbaz
# STATUS: 0
# STDIO <<EOM
# foo
# bar
# baz
# EOM
ok 8 - RUNS echo -e 'foo\nbar\nbaz'
# grep -q bar
ok 9 - GREP bar
# grep -q bar
ok 10 - OGREP bar
# \! grep -q .
ok 11 - NEGREP . # verify empty
ok 12 - DIFF <<EOF
# RUNNING: false
# STATUS: 1
# STDIO <<EOM
# EOM
not ok 13 - TODO RUNS false # TODO Test Know to fail
Rename it to use a .t extension and put it in a t subdirectory, and you can use prove(1) (part of Perl) to run it:
$ prove
t/test.t .. ok
All tests successful.
Files=1, Tests=13, 0 wallclock secs ( 0.03 usr 0.01 sys + 0.11 cusr 0.16 csys = 0.31 CPU)
Result: PASS
Set OSHT_JUNIT or pass -j to produce JUnit output. JUnit can also be combined with prove(1).
I have used this library both testing functions by sourcing their files and then running assertions with IS/OK and their negatives, and scripts by using RUN/NRUN. For me, this framework provides the most gain for the least overhead.
Why do you say that it's "hard" to test Bash scripts?
What's wrong with test wrappers like the following?
#!/bin/bash
set -e
errors=0
results=$($script_under_test $args<<ENDTSTDATA
# inputs
# go
# here
#
ENDTSTDATA
)
[ "$?" -ne 0 ] || {
echo "Test returned error code $?" 2>&1
let errors+=1
}
echo "$results" | grep -q $expected1 || {
echo "Test Failed. Expected $expected1"
let errors+=1
}
# And so on, et cetera, ad infinitum, ad nauseum
[ "$errors" -gt 0 ] && {
echo "There were $errors errors found"
exit 1
}
Epoxy is a Bash test framework I designed mainly for testing other software, but I use it to test Bash modules as well, including itself and Carton.
The main advantages are relatively low coding overhead, unlimited assertion nesting and flexible selection of assertions to verify.
I made a presentation comparing it to BeakerLib - a framework used by some at Red Hat.
Give assert.sh a try:
source "./assert.sh"
local expected actual
expected="Hello"
actual="World!"
assert_eq "$expected" "$actual" "not equivalent!"
# => x Hello == World :: not equivalent!
I quite like shell2junit, a utility to generate JUnit-like output from Bash script tests. This is useful because the report generated can then be read by continuous integration systems, such as the JUnit plug-ins for Jenkins and Bamboo.
While shell2junit doesn't provide the comprehensive Bash scripting framework like shunit2, it does allow you have nice reporting of the test results.
Try bashtest. It’s simple way to test your scripts. For example, you have do-some-work.sh which changes some configuration files. For example, add a new line, PASSWORD = 'XXXXX', to configuration file /etc/my.cfg.
You write Bash commands line by line and then check output.
Install:
pip3 install bashtest
Create tests is a just writing bash commands.
File test-do-some-work.bashtest:
# Run the script
$ ./do-some-work.sh > /dev/null
# Testing that the line "PASSWORD = 'XXXXX'" is in the file /etc/my.cfg
$ grep -Fxq "PASSWORD = 'XXXXX'" /etc/my.cfg && echo "YES"
YES
Run tests:
bashtest *.bashtest
You can find some examples here and here.
Maybe this can be used, or contributed to:
https://thorsteinssonh.github.io/bash_test_tools/
It is intended to write results in the TAP protocol which I imagine is good for CI and is good for those that want shell environments. I imagine some things run in shell environments, so some might argue it should be tested in their shell environment.
I’ve tried a lot of the solutions presented here, but I found most of them too bulky and hard to use, so I built my own little testing framework: https://github.com/SnacOverflow/t-bash
It’s just one file in the repository that you can simply run directly, with a basic set of JUnit style asserts.
I’ve used it professionally in several internal projects and were able to make our Bash scripts super stable and regression resistant.
Create a test mytest.sh. It calls your script with specific inputs.
Create a text file expected/mytest.stdout. It contains the expected output for the test.
Commit them to version control (if any).
Run the test and redirect the output to another file:
mytest.sh > actual/mytest.stdout
Then just us a text diff tool to see where the results deviate. I think you can just do
diff expected/mytest.stdout actual/mytest.stdout
Take a look at Outthentic. It is simple, extensible by many languages (Perl, Python, Ruby, and Bash on choice) and cross platform (Linux and Windows) framework to test any command line applications.
