5

Some of the tasks I run in Airflow occasionally fail for various reasons. Some I catch and others are new and aren't excepted. I want to get exit code 1 in order to receive a notification of such failures.

Below is an example of an error I didn't except, is there a way to have ValueError yield exit code 1? 

[2019-06-13 12:56:13,630] {bash_operator.py:127} INFO - ValueError: year 43631 is out of range
[2019-06-13 12:56:13,870] {bash_operator.py:127} INFO - 2019-06-13 12:56:13,869 - ODL - INFO - Closing db connection
[2019-06-13 12:56:13,870] {bash_operator.py:127} INFO - 2019-06-13 12:56:13,870 - ODL - INFO - End
[2019-06-13 12:56:13,990] {bash_operator.py:131} INFO - Command exited with return code 0
[2019-06-13 12:56:17,556] {logging_mixin.py:95} INFO - [2019-06-13 12:56:17,555] {jobs.py:2562} INFO - Task exited with return code 0

I also have other bits of code where the try block uses sys.exit(1). However, it is not raised, do I need to raise this error in order to exit with 1?

I've tried using sys.exit(1) in my try blocks. 

        if count==0:
            self._logger.error("Missing data, nothing was loaded")
            sys.exit(1)
        else:
            self._logger.info("Data is present, safe to load to prod!")

I expected to get the following in order to be notified of the failure. 

2019-06-13 12:56:17,556] {logging_mixin.py:95} INFO - [2019-06-13 12:56:17,555] {jobs.py:2562} INFO - Task exited with return code 1
papacoolaid666
  • 85
  • 9
Sasha Podoprigora
  • 53
  • 1
  • 4

1 Answers1

2

It looks like you're using a BashOperator to call your Python script. This means you'll want to use something like set -e to ensure Bash stops execution when it hits the non-zero code on your Python command and it exits itself with the same code, see What does set -e mean in a bash script? for more details. Then this should fail the task as you'd expect.

Daniel Huang
  • 6,238
  • 34
  • 33