3

I know there are a few questions on here and around the internet regarding parallel python builds using setuptools. Yet, none provide a solution. It's also worth mentioning that most I come across are 5+ years old.

It seems that I can achieve my desired parallelization during install through:

>> python setup.py build -j 4
>> python setup.py install

But this doesn't work for editable installations, i.e. for development. The editable install requires calling python setup.py develop instead of the above 2 commands. However, there is no -j flag for develop. If I try to call it after the build, for example,

>> python setup.py build -j 4
>> python setup.py develop

I get the error:

running develop
running egg_info
creating My_Package.egg-info
writing My_Package.egg-info/PKG-INFO
writing dependency_links to My_Package.egg-info/dependency_links.txt
writing top-level names to My_Package.egg-info/top_level.txt
writing manifest file 'My_Package.egg-info/SOURCES.txt'
reading manifest file 'My_Package.egg-info/SOURCES.txt'
writing manifest file 'My_Package.egg-info/SOURCES.txt'
running build_ext
copying build/lib.linux-x86_64-3.7_my_package_ext/_enums.cpython-37m-x86_64-linux-gnu.so _my_package_ext
error: could not create _my_package_ext/_enums.cpython-37m-x86_64-linux-gnu.so': No such file or directory

Additionally, it is recommended to use pip install -e . --user instead of python setup.py develop anyway. If I try to call that after the parallel build, I get:

>> python setup.py build -j 4
>> pip install -e . --user

Running setup.py (path:/package/setup.py) egg_info for package from file:///package
    Running command python setup.py egg_info
    running egg_info
    writing My-Package.egg-info/PKG-INFO
    writing dependency_links to My-Package.egg-info/dependency_links.txt
    writing top-level names to My-Package.egg-info/top_level.txt
    reading manifest file 'My-Package.egg-info/SOURCES.txt'
    writing manifest file 'My-Package.egg-info/SOURCES.txt'
  Source in /package has version 0.0, which satisfies requirement My-Package==0.0 from file:///package
Installing collected packages: My-Package
  Running setup.py develop for My-Package
    Running command /usr/bin/python3 -c "import setuptools, tokenize;__file__='/package/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --user --prefix=
    running develop
    running egg_info
    writing My-Package.egg-info/PKG-INFO
    writing dependency_links to My-Package.egg-info/dependency_links.txt
    writing top-level names to My-Package.egg-info/top_level.txt
    reading manifest file 'My-Package.egg-info/SOURCES.txt'
    writing manifest file 'My-Package.egg-info/SOURCES.txt'
    running build_ext
    copying build/lib.linux-x86_64-3.6/_my_package_ext/_enums.cpython-36m-x86_64-linux-gnu.so -> _my_package_ext
    error: could not create '_my_package_ext/_enums.cpython-36m-x86_64-linux-gnu.so': No such file or directory
Cleaning up...
Command "/usr/bin/python3 -c "import setuptools, tokenize;__file__='/package/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --user --prefix=" failed with error code 1 in /package/

Does anyone know of a good way to get the parallelized build in an editable install? Ideally, it would be though the pip install -e ... formulations, but really anything would be better than nothing.

marcman
  • 3,233
  • 4
  • 36
  • 71
  • 1
    Somewhat related, maybe some insight to gather from there: https://discuss.python.org/t/the-difference-between-python-setup-py-build-ext-i-and-pip-install-e/3716 – sinoroc Mar 19 '20 at 14:56
  • 2
    `python setup.py build -j5 develop` should work, or else add a [mcve]. – hoefling Mar 19 '20 at 19:28
  • 1
    For dev installs with `pip`, `pip install --editable . --global-option="build_ext" --global-option="-j5"` should do it. – hoefling Mar 19 '20 at 19:30
  • @hoefling: Any docs on why `python setup.py build -j 5 develop` would work, but doing them as separate commands does not? BTW it did not work for me--fails with the same error as the OP. I'll see about a minimal reproducible example – marcman Mar 19 '20 at 21:54
  • The command listed in your question should also work, the error has smth to do with your setup script/project layout. – hoefling Mar 20 '20 at 13:33
  • 1
    `distutils/command/build_ext.py` only parallelizes over each *extension* in a package, rather than every source file in the build like a proper Makefile would. – jbarlow Jun 27 '20 at 09:30

0 Answers0