How to continue a task when Fabric receives an error

Question

When I define a task to run on several remote servers, if the task runs on server one and exits with an error, Fabric will stop and abort the task. But I want to make fabric ignore the error and run the task on the next server. How can I make it do this?

For example:

$ fab site1_service_gw
[site1rpt1] Executing task 'site1_service_gw'

[site1fep1] run: echo 'Nm123!@#' | sudo -S route
[site1fep1] err:
[site1fep1] err: We trust you have received the usual lecture from the local System
[site1fep1] err: Administrator. It usually boils down to these three things:
[site1fep1] err:
[site1fep1] err:     #1) Respect the privacy of others.
[site1fep1] err:     #2) Think before you type.
[site1fep1] err:     #3) With great power comes great responsibility.
[site1fep1] err: root's password:
[site1fep1] err: sudo: route: command not found

Fatal error: run() encountered an error (return code 1) while executing 'echo 'Nm123!@#' | sudo -S route '

Aborting.

Answer 1

From the docs:

... Fabric defaults to a “fail-fast” behavior pattern: if anything goes wrong, such as a remote program returning a nonzero return value or your fabfile’s Python code encountering an exception, execution will halt immediately.

This is typically the desired behavior, but there are many exceptions to the rule, so Fabric provides env.warn_only, a Boolean setting. It defaults to False, meaning an error condition will result in the program aborting immediately. However, if env.warn_only is set to True at the time of failure – with, say, the settings context manager – Fabric will emit a warning message but continue executing.

Looks like you can exercise fine-grained control over where errors are ignored by using the settings context manager, something like so:

from fabric.api import settings

sudo('mkdir tmp') # can't fail
with settings(warn_only=True):
    sudo('touch tmp/test') # can fail
sudo('rm tmp') # can't fail

Answer 2

As of Fabric 1.5, there is a ContextManager that makes this easier:

from fabric.api import sudo, warn_only

with warn_only():
    sudo('mkdir foo')

Update: I re-confirmed that this works in ipython using the following code.

from fabric.api import local, warn_only

#aborted with SystemExit after 'bad command'
local('bad command'); local('bad command 2')

#executes both commands, printing errors for each
with warn_only():
    local('bad command'); local('bad command 2')

Answer 3

You can also set the entire script's warn_only setting to be true with

def local():
    env.warn_only = True

Answer 4

You should set the abort_exception environment variable and catch the exception.

For example:

from fabric.api        import env
from fabric.operations import sudo

class FabricException(Exception):
    pass

env.abort_exception = FabricException
# ... set up the rest of the environment...

try:
    sudo('reboot')
except FabricException:
    pass  # This is expected, we can continue.

You can also set it in a with block. See the documentation here.

Answer 5

In Fabric 2.x you can just use invoke's run with the warn=True argument. Anyway, invoke is a dependency of Fabric 2.x:

from invoke import run
run('bad command', warn=True)

From within a task:

from invoke import task

@task
def my_task(c):
    c.run('bad command', warn=True)

Answer 6

In Fabric 1.3.2 at least, you can recover the exception by catching the SystemExit exception. That's helpful if you have more than one command to run in a batch (like a deploy) and want to cleanup if one of them fails.

Answer 7

In my case, on Fabric >= 1.4 this answer was the correct one.

You can skip bad hosts by adding this:

env.skip_bad_hosts = True

Or passing the --skip-bad-hosts flag/