Handling python error in bash script with no error codes

Question

I am running a python cli tool, Papermill. I am running jupyter notebooks and want to be able to detect assertionErrors when the notebook are ran. When I run papermill from a bash script I always get a 0 exit code, even when there is a python error in the notebook.

I've found methods for detecting the exit code which work, but python errors are not throwing any codes

For example: in my nbTest.sh I am running the notebook and looking for an error

papermill python_notebooks/testing/$d python_notebooks/testing/results/$d.results.ipynb || True
if [ $? -eq 0 ]
then
  echo "Successfully executed script"
else
  # Redirect stdout from echo command to stderr.
  echo "Script exited with error." 
fi
done

and as output I am getting:

.
.
.
Input Notebook:  python_notebooks/testing/paperTest.ipynb
Output Notebook: python_notebooks/testing/results/paperTest.ipynb.results.ipynb
 83%|████████▎ | 5/6 [00:00<00:00,  1.97it/s]
Traceback (most recent call last):
 ...
---------------------------------------------------------------------------
Exception encountered at "In [5]":
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-5-d941c6663321> in <module>()
----> 1 assert passed == False

AssertionError: 
Successfully executed script

I would expect an exit code of 1 but python failures aren't handled

Isn't it better to handle error inside your python code? Something like `try: assert True assert 7 == 7 assert 1 == 2 # many more statements like this except AssertionError: _, _, tb = sys.exc_info() traceback.print_tb(tb) # Fixed format tb_info = traceback.extract_tb(tb) filename, line, func, text = tb_info[-1] print('An error occurred on line {} in statement {}'.format(line, text)) exit(1)` — Irfanuddin, Aug 14 '19 at 18:43

score 0 · Answer 1 · answered Oct 05 '19 at 01:23

Part of the beauty of the Jupyter and papermill combo is the errors are preserved in the notebook and location of where they occurred. This is very helpful when running multiple ETL jobs at once. If you also want access to the errors outside of the notebook, I would write the specific errors out to a errors.txt file from inside of a try catch. Then your script can finish by reading off the errors, if any.

Various error logging techniques are discussed in this post.

Handling python error in bash script with no error codes

1 Answers1