How to log from Python Luigi

Question

I am trying to build a strategy to log from Luigi in such a way that there is a configurable list of outputs including stdout and a custom list of files. I would like to be able to set the logging level at runtime. Our system uses Luigi to call spark from Jenkins. Thank you in advance.

score 15 · Accepted Answer · answered Mar 31 '19 at 17:24

Inside any of the Task class methods, you can do:

class Agg(luigi.Task):
  _date = luigi.DateParameter()

  def output(self):
    return luigi.LocalTarget("file_%.txt" % self._date)

  def run(self):
    # Use the luigi-interface to log to console
    logger = logging.getLogger('luigi-interface')
    logger.info("Running --> Agg.Task")

score 0 · Answer 2 · edited Sep 17 '17 at 17:29

0

Have you checked the logging_conf_file parameter of the configuration? you can set up there all your configuration regarding logging using Python's standard logging mechanism.

For some examples see:

edited Sep 17 '17 at 17:29

Brian Burns

20,575
8
83
77

answered Jun 20 '16 at 10:24

mfcabrera

781
10
26

6

It would be really great if you could provide solutions to answers rather than links to other issue pages. The other issues and their contents are just as unhelpful. – elliotwesoff Aug 21 '18 at 14:22

How to log from Python Luigi

2 Answers2