0

Look at class ATask

class ATask(luigi.Task):
    config = luigi.Parameter()

    def requires(self):
         # Some Tasks maybe

    def output(self):
        return luigi.LocalTarget("A.txt")

    def run(self):
        with open("A.txt", "w") as f:
            f.write("Complete")

Now look at class BTask

class BTask(luigi.Task):
    config = luigi.Parameter()

    def requires(self):
         return ATask(config = self.config)

    def output(self):
        return luigi.LocalTarget("B.txt")

    def run(self):
        with open("B.txt", "w") as f:
            f.write("Complete")

Question is there is a chance that while TaskA running and start write "A.txt" taskB will start before taskA finishing writing?

The second is that if I start execution like

luigi.build([BTask(config=some_config)], local_scheduler=True )

And if this pipilene fail inside - Could I somehow to know outside about this like return value of luigi.build or smth else?

Anton
  • 73
  • 8

1 Answers1

1

  1. No, luigi won't start executing TaskB until TaskA has finished (ie, until it has finished writing the target file)

  2. If you want to get a detailed response for luigi.build in case of error, you must pass an extra keyword argument: detailed_summary=True to build/run methods and then access the summary_text, this way:

     luigi_run_result = luigi.build(..., detailed_summary=True)
     print(luigi_run_result.summary_text)

For details on that, please read Response of luigi.build()/luigi.run() in Luigi documentation.

Also, you may be interested in this answer about how to access the error / exception: https://stackoverflow.com/a/33396642/3219121

matagus
  • 6,136
  • 2
  • 26
  • 39
  • Thank you, very useful for me indeed. But how, luigi will internally control starting of taskB? Because if "A.txt" will be already on hard drive before luigi.build - luigi will start taskB without starting taskA, as I know – Anton Feb 15 '22 at 05:04
  • luigi.build([Task(config=config)], local_scheduler=True, detailed_summary=True) and I got exception "unknown parameter detailed_summary". I didn't find in search such problems. Do you have idea what happened ? – Anton Feb 15 '22 at 08:55