saltstack testing: render as each host in turn

Question

We test saltstack scripts on vagrant boxes. It sometimes happens that while testing changes for host A, we inadvertently cause a problem for host B.

I'd like to write a script that will at least render (call show_sls) as if the minion id were each host in turn. This will remove at least one class of error.

I can think of some hacky ways of doing this, like changing the contents of /etc/salt/minion_id on each iteration. But are there better ways?

I know far too little about how to test salt scripts well. All suggestions most welcome.

Pseudo-code example

To make this somewhat clearer, what I'd like might resemble this:

my_minions='a.example.com b.example.com c.example.com'
for minion_id in $my_minions; do
    sudo salt-call --local --minion="$minion_id" state.show_sls > /dev/null
    if [ $? != 0 ]; then
        echo "$minion_id does not render"
    fi
done

Answer 1

You can use salt test interface to perform dry-run on all the minions by setting test=True option to the state along with --retcode-passthrough flag which will exit with the salt call retcode and not the salt binary retcode, and check the retcode which is returned to check if the state will be rendered successfully, i.e,

salt-call state.sls teststate test=True --retcode-passthrough; echo $?

will print non-zero exit code if any state is unsuccessful. To get list of minions you can loop through /etc/salt/pki/master/minions

(You can also set test to true by default in minion configuration file, then states will default to being executed in test mode.)

2nd approach could be setting Failhard global option in your master which will immediately stop all state execution and throw an error. You can run it in a same way as above, in this case execution will stop immediately after first failure, and you will have your retcode.

You might want to look at kitchen-salt as well,

The goal of this kitchen-salt is to make it easy to test Salt States or Formulas independently of a production environment. It allows for doing quick checks of states and to make sure that upstream changes in packages will not affect deployments. By using platforms, users can run checks on their states against the environment they are running in production as well as checking future releases of distributions before doing major upgrades. It is also possible to test Salt States against multiple versions of Salt to make sure there are no major regressions.

I would also like to mention that salt has no proper exit codes yet, and there are many issues still open on this subject. If you want to check the output manually, you can use this python script which is mentioned on Track failure of command on a Salt minion