How to catch bash script error, which is run through a python script?

Question

I'm using Airflow and i have some python scripts, one of it launch sh script (1 or many). For airflow even if this bash script return an error, it considers that the task is success.

I would like to return the bash error in my python script as airflow will notice.

import glob
import os
from os import path
import subprocess

odsPath = '/apps/data/02_OS/'


for odsFolder in os.listdir(odsPath):
    if(odsFolder in fileNames):
        newPath = odsPath+odsFolder
        print('Script path: ', newPath)
        for foldFiles in os.listdir(newPath):
            if(foldFiles.endswith('sh')):
                print('Sh script can be launch: ', foldFiles)
            if(foldFiles.startswith('run') and foldFiles.endswith('.sh')):
                os.system('sh ' + newPath+'/'+foldFiles)
                print('Script: '+foldFiles+' executed')

In this case, there are bash errors, like this for example 'Number of parameters is incorrect'

Thanks for help :)

What have you tried? You want to capture the 'STDERR' output (or maybe 'STDOUT' too if the scripts are naughty and use the wrong output stream) https://en.wikipedia.org/wiki/Standard_streams https://docs.python.org/2/library/subprocess.html — Pocketsand, Feb 21 '20 at 15:23

score 3 · Accepted Answer · answered Feb 21 '20 at 15:25

os.system() runs a command completely outside of Python's control, though you can examine its return code and raise an error if there is a failure. As suggested in the os.system documentation, a better approach which gives you more control and fewer moving parts is subprocess:

subprocess.run(['sh', os.path.join(newPath,foldFiles)],
    check=True)

The check=True causes a subprocess.CalledProcessError to be raised if the subprocess fails. See further Running Bash commands in Python

How to catch bash script error, which is run through a python script?

1 Answers1