Luigi Pipelining : No module named pwd in Windows

Question

I am trying to execute the tutorial given in https://marcobonzanini.com/2015/10/24/building-data-pipelines-with-python-and-luigi/.

I am able to run the program on its own using local scheduler, giving me:

Scheduled 2 tasks of which:
* 2 ran successfully:
    - 1 PrintNumbers(n=1000)
    - 1 SquaredNumbers(n=1000)

This progress looks :) because there were no failed tasks or missing external de
pendencies

===== Luigi Execution Summary =====

However, to try the visualization on the server, when I try to run luigid --background, it throws me an error saying I dont have pwd module. I cannot find a pwd module using pip for windows.

  File "c:\users\alex\appdata\local\continuum\anaconda3\lib\site-packages
\luigi\process.py", line 79, in daemonize
    import daemon
  File "c:\users\alex\appdata\local\continuum\anaconda3\lib\site-packages
\daemon\__init__.py", line 42, in <module>
    from .daemon import DaemonContext
  File "c:\users\alex\appdata\local\continuum\anaconda3\lib\site-packages
\daemon\daemon.py", line 25, in <module>
    import pwd
ModuleNotFoundError: No module named 'pwd'

I am working in Anaconda Spyder with Python 3.6

score 5 · Answer 1 · answered Aug 22 '18 at 17:02

5

I was able to fix this by installing python-daemon==2.1.2 If you already have python-daemon, try downgrading to version 2.1.2 Do this before install luigi.

Example: pip install python-daemon==2.1.2 then pip install luigi.

answered Aug 22 '18 at 17:02

Christian

51
1
2

score 4 · Answer 2 · answered Jul 23 '18 at 21:54

4

For some reason, if you dont use the --background parameter on windows it will start just fine

just write luigid in cmd

answered Jul 23 '18 at 21:54

Steven G

16,244
8
53
77

mikeviescas · Answer 3 · 2018-07-26T19:22:49.797

The base problem here is that luigid --background is trying to spawn a python-daemon which is a unix-specific thing.

See section titled "The luigid server" here: http://luigi.readthedocs.io/en/stable/central_scheduler.html

Specifically:

Note that this requires python-daemon. By default, the server starts on AF_INET and AF_INET6 port 8082 (which can be changed with the --port flag) and listens on all IPs. (To use an AF_UNIX socket use the --unix-socket flag)

This existing stack overflow answer provides more detail:

How to start daemon process from python on windows?

Options I see here are:

Log a request with Luigi on github to improve their windows support to spawn Luigid as a windows process for the --background switch
Run a virtual machine with a proper Unix OS in it on Windows and run your Luigi pipelines there.
Follow Steven G's suggestion and run luigid in a separate command prompt

To reproduce the root cause of this issue, open a python prompt in windows and type:

>>import daemon

Traceback (most recent call last): File "", line 1, in File "C:\Anaconda3\lib\site-packages\daemon__init__.py", line 42, in from .daemon import DaemonContext File "C:\Anaconda3\lib\site-packages\daemon\daemon.py", line 25, in import pwd ModuleNotFoundError: No module named 'pwd'

Luigi Pipelining : No module named pwd in Windows

3 Answers3