2

Issue

I am trying to get setup.py test working with a package that uses multiprocessing on Windows. I have the following scenario:

  • A regular python package folder structure
  • with a package
  • a test folder and
  • a setup.py.

Running

python -m unittest

from the top directory executes my tests without complaints. However, when calling

python setup.py test

I get the well known freeze_support issue (see several questions here on SO):

RuntimeError: 
    An attempt has been made to start a new process before the
    current process has finished its bootstrapping phase.
    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:
        if __name__ == '__main__':
            freeze_support()
            ...
    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

However, following where to put freeze_support() in a Python script? I actually have the described guards in place in all my files, possibly in way too many.

Question 1: What do I have to change in my project for setup.py test to work with my multiprocessing using function?

Question 2: Which of my placed freeze_support() are actually needed? As far as I know, this is normally only needed when compiling a frozen executable.

MWE

Folder structure

project
│   setup.py
│
└─── my_package
│    │   __init__.py
│    │   my_package.py
│   
└─── test
│    │   __init__.py
│    │   test_it.py

The __init__.py files are empty, the other ones are:

my_package.py

import multiprocessing

def say_hi(arg):
    print('hi')

def main():
    pool = multiprocessing.Pool(1)
    pool.map(say_hi, range(1))

if __name__ == '__main__':
    multiprocessing.freeze_support()
    main()

test_it.py

import multiprocessing
import unittest

from my_package import my_package

class TestIt(unittest.TestCase):
    def test_say_hi(self):
        my_package.main()

if __name__ == '__main__':
    multiprocessing.freeze_support()

setup.py

#!/usr/bin/env python

import setuptools
import multiprocessing

setuptools.setup(
    name = "my_package"
)

if __name__ == '__main__':
    multiprocessing.freeze_support()
Community
  • 1
  • 1
NOhs
  • 2,780
  • 3
  • 25
  • 59

2 Answers2

4

I encountered the similar problem. I found that the new process created by multiprocessing calls setuptools.setup method when the test is executed via setup.py and this affects something. (I'm not clear about the root cause...)

In my case, this setup.py resolved that.

#!/usr/bin/env python

import setuptools

if __name__ == '__main__':
    setuptools.setup(
        name = "my_package",
        ....,
    )
user1397507
  • 56
  • 3
1

As previously mentioned, I think this program runs if setuptools.setup is executed only when __name__ == ‘main’ holds.

Root cause

When you spawn the process with multiprocessing, modules are imported, setting __name__ attributes to be “__mp_main__”, (not __main__).

This is a code snippet from _fixup_main_from_path in multiprocessing.spawn.py

# If the parent process has sent a path through rather than a module
# name we assume it is an executable script that may contain
# non-main code that needs to be executed
old_main_modules.append(current_main)
main_module = types.ModuleType("__mp_main__")
main_content = runpy.run_path(main_path,
                              run_name="__mp_main__")
main_module.__dict__.update(main_content)
sys.modules['__main__'] = sys.modules['__mp_main__'] = main_module

Therefore, if setuptools.setup is executed at the top level of module, infinite recursion occurs.

user9809355
  • 11
  • 1