Can I route Luigi pipeline errors to Sentry?

Question

My team uses Sentry to track errors, and so I would prefer to not use Luigi's built-in email capabilities to keep all our reporting in one place.

This is how I have it currently set up, and it seems to be skipping Sentry entirely:

if __name__ == '__main__':
    try:
        luigi.run()
    except Exception as e:
        client = Client(
            ***
        )
        client.captureException(tags={
            sys.argv[0]
        })
        logger.critical('Error occurred: {e}'.format(e=e))
        raise

score 5 · Accepted Answer · edited May 23 '17 at 12:01

I think it should be possible if you declare a callback to the failure event and do the sentry tracking stuff there:

import luigi

@luigi.Task.event_handler(luigi.Event.FAILURE)
def mourn_failure(task, exception):
    client = Client(
        ***
    )
    # we also include extra context for the current task
    client.captureException(
       (type(e), e.message, e.traceback),
       extra=dict(task=repr(task))
    )
    logger.critical('Error occurred: {e}'.format(e=exception))


if __name__ == '__main__':
    luigi.run()

NOTICE that e.traceback will only work on python 3+ as explained in this answer.

Can I route Luigi pipeline errors to Sentry?

1 Answers1