shared volume with luigi docker server

Question

I am trying to experiment a bit with docker and luigi https://hub.docker.com/r/spotify/luigi

I created a docker container with spotify/luigi. I am quite new to this and this image does not seem to be accessible through the console.

I created a shared volume and mapped it to /luigi/share/ for the container

import luigi
import time

class HelloWorld(luigi.Task):
    def requires(self):
        return None
    def output(self):
        return luigi.LocalTarget('/luigi/share/helloworld3.txt')
    def run(self):
        time.sleep(1)
        with self.output().open('w') as outfile:
            outfile.write('Hello World!\n')
        time.sleep(1)

class NameSubstituter(luigi.Task):
    name = luigi.Parameter()

    def requires(self):
        return HelloWorld()
    def output(self):
        return luigi.LocalTarget(self.input().path + '.name_' + self.name)
    def run(self):
        time.sleep(1)
        with self.input().open() as infile, self.output().open('w') as outfile:
            text = infile.read()
            text = text.replace('World', self.name)
            outfile.write(text)
        time.sleep(1)

if __name__ == '__main__':
    luigi.run()

This is the example Code I run with python test.py --scheduler-host 192.168.178.48 NameSubstituter

When I now look into another container with the shared volume there is no text file created.

I am a little lost here...

Thank you in advance

score 0 · Answer 1 · answered Oct 10 '19 at 18:29

When you run test.py directly from Python, you aren't actually running anything through docker. You need to follow the instructions actually set up your container with your test.py inside and then run it from docker. This link might have more information: How to run my python script on docker?

shared volume with luigi docker server

1 Answers1