2

I am using Glue bookmarking to process data. My job is scheduled every day, but can also be launch "manually". Since I use bookmarks, sometimes the Glue job can start without having new data to process, the read dataframe is then empty. In this case, I want to end my job properly because it has nothing to do. I tried:

if df.rdd.isEmpty():
    job.commit()
    sys.exit(0)

However, my job terminate in error with SystemExit: 0.

How to end the job with success?

Jérémy
  • 1,790
  • 1
  • 24
  • 40
  • My question is a fork of the not working answer to https://stackoverflow.com/questions/67028388/how-to-stop-exit-a-aws-glue-job-pyspark – Jérémy Sep 22 '21 at 09:27

2 Answers2

5

Just using os._exit() doesn't work in Glue Version 3.0

To exit a job gracefully after some conditions have been met, use:

import os
import sys
.
. # Your Glue Job Code
.
logger.info("Existing job gracefully...") # Or a simple print; print("...")
job.commit() # Only necessary if you are loading data from s3 and you have job bookmarks enabled.
os._exit(0) # Using a 0 status code throws no exception so your job completes with a succeeded status.

But, if you want to exit with an error use:

sys.exit("Error Message...") # this will exit with an error message that will be displayed on the Glue UI -- Run Details --, and the job would have a status of failed.
jay-dono
  • 51
  • 1
  • 2
3

After some test, I discovered from @Glyph's answer that :

os._exit() terminates immediately at the C level and does not perform any of the normal tear-downs of the interpreter.

Which is exactly what I was looking for. The final solution is:

import os

if df.rdd.isEmpty():
    job.commit()
    os._exit()
Jérémy
  • 1,790
  • 1
  • 24
  • 40
  • 2
    In my case `os._exit()` was failling with error **_TypeError: _exit() missing required argument 'status' (pos 1)_** . I had to provide one of the status of exit. For success you can provide 0 as a status `os._exit(0)` . For other status this [link](https://docs.python.org/3/library/os.html#os._exit) can be referred. – Yadav Sep 23 '22 at 06:25