Kill next command in pipeline on failure

Question

I have a streaming backup script which I'm running as follows:

./backup_script.sh | aws s3 cp - s3://bucket/path/to/backup

The aws command streams stdin to cloud storage in an atomic way. If the process is interrupted without an EOF, the upload is aborted.

I want the aws process to be killed if ./backup_script.sh exits with a non-zero exit code.

Any bash trick for doing this?

EDIT: You can test your solution with this script:

#!/usr/bin/env python
import signal
import sys
import functools

def signal_handler(signame, signum, frame):
    print "Got {}".format(signame)
    sys.exit(0)

signal.signal(signal.SIGTERM, functools.partial(signal_handler, 'TERM'))
signal.signal(signal.SIGINT, functools.partial(signal_handler, 'INT'))

for i in sys.stdin:
    pass

print "Got EOF"

Example:

$ grep --bla | ./sigoreof.py
grep: unrecognized option `--bla'
usage: grep [-abcDEFGHhIiJLlmnOoqRSsUVvwxZ] [-A num] [-B num] [-C[num]]
    [-e pattern] [-f file] [--binary-files=value] [--color=when]
    [--context[=num]] [--directories=action] [--label] [--line-buffered]
    [--null] [pattern] [file ...]
Got EOF

I want ./sigoreof.py to be terminated with a signal.

Answer 1

Adopting/correcting a solution originally given by @Dummy00001:

mkfifo aws.fifo
exec 3<>aws.fifo # open the FIFO read/write *in the shell itself*
aws s3 cp - s3://bucket/path/to/backup <aws.fifo 3>&- & aws_pid=$!
rm aws.fifo # everyone who needs a handle already has one; can remove the directory entry

if ./backup_script.sh >&3 3>&-; then
    exec 3>&-       # success: close the FIFO and let AWS exit successfully
    wait "$aws_pid"
else
    kill "$aws_pid" # send a SIGTERM...
    wait "$aws_pid" # wait for the process to die...
    exec 3>&-       # only close the write end *after* the process is dead
fi

---

Important points:

• The shell opens the FIFO r/w to avoid blocking (opening for write only would block for a reader; this could also be avoided by invoking the reader [that is, the s3 command] in the background prior to the exec opening the write side).
• The write end of the FIFO is held by the script itself, so the read end never hits end-of-file until after the script intentionally closes it.
• The aws command's handle on the write end of the FIFO is explicitly closed (3<&-), so it doesn't hold itself open (in which case the exec 3>&- done in the parent would not successfully allow it to finish reading and exit).

Answer 2

backup_script.sh should have a non-zero exit status if there is an error, so you script should look something like:

if ./backup_script.sh > output.txt; then
    aws s3 cp output.txt s3://bucket/path/to/backup
fi
rm -f output.txt

A pipe isn't really appropriate here.

---

If you really need to conserve disk space locally, you'll have to "reverse" the upload; either remove the uploaded file in the event of an error in backup_script.sh, or upload to a temporary location, then move that to the final path once you've determined that the backup has succeeded.

(For simplicity, I'm ignoring the fact that by letting aws exit on its own in the event of an error, you may be uploading more of the partial backup than you need to. See Charles Duffy's answer for a more bandwidth-efficient approach.)

After starting the backup process with

mkfifo data
./backup_script.sh > data & writer_pid=$!

use one of the following to upload the data.

# Upload and remove if there was an error
aws s3 cp - s3://bucket/path/to/backup < data &

if ! wait $writer_pid; then
    aws s3 rm s3://bucket/path/to/backup
fi

or

# Upload to a temporary file and move it into place
# once you know the backup succeeded.
aws s3 cp - s3://bucket/path/to/backup.tmp < data &

if wait $writer_pid; then
    aws s3 mv s3://bucket/path/to/backup.tmp s3://bucket/path/to/backup
else
    aws s3 rm s3://bucket/path/to/backup
fi

Answer 3

A short script which uses process substitution instead of named pipes would be:

#!/bin/bash

exec 4> >( ./second-process.sh )
./first-process.sh >&4  &
if ! wait $! ; then echo "error in first process" >&2; kill 0; wait; fi

It works much like with a fifo, basically using the fd as the information carrier for the IPC instead of a file name.

Two remarks: I wasn't sure whether it's necessary to close fd 4 ; I would assume that upon script exit the shell closes all open files.

And I couldn't figure out how to obtain the PID of the process in the process substitution (anybody? at least on my cygwin the usual $! didn't work.) Therefore I resorted to killing all processes in the group, which may not be desirable (but I'm not entirely sure about the semantics).

Answer 4

I think you need to spawn both processes from a third one and either use the named pipe approach from Lynch in the post mentioned by @tourism (further below in the answers); or keep piping directly but re-write backup_script.sh such that it stays alive in the error case, keeping stdout open. backup_script.sh would have to signal the error condition to the calling process (e.g. by sending a SIGUSR to the parent process ID), which in turn first kills the aws process (leading to an atomic abort) and only then backup_script.sh, unless it exited already because of the broken pipe.

Answer 5

I had a similar situation: a shell script contained a pipeline that used one of its own functions and that function wanted to be able to effect termination. A simple contrived example that finds and displays a file:

#!/bin/sh
a() { find . -maxdepth 1 -name "$1" -print -quit | grep . || exit 101; }
a "$1" | cat
echo done

Here, the function a needs to be able to effect termination which it tries to do by calling exit. However, when invoked through a pipeline (line 3), it only terminates its own (subshell) process. In the example, the done message still appears.

One way to work around this is to detect when in a subshell and send a signal to the parent:

#!/bin/sh
die() { [[ $$ == $(exec sh -c 'echo $PPID') ]] && exit $1 || kill $$; }
a() { find . -maxdepth 1 -name "$1" -print -quit | grep . || die 101; }
a "$1" | cat
echo done

When in a subshell the $$ is the pid of the parent and the construct $(exec sh -c 'echo $PPID') is a shell-agnostic way to obtain the pid of the subprocess. If using bash then this can be replaced by $BASHPID.

If the subprocess pid and $$ differ then it sends a SIGTERM signal to the parent (kill $$) instead of calling exit.

The given exit status (101) isn't propagated by kill so the script exits with a status of 143 (which is 128+15 where 15 is the id of SIGTERM).