70

I found that pip only use single core when it compiles packages. Since some python packages takes some time to build using pip, I'd like to utilize multicore on the machine. When using Makefile, I can do that like following command:

make -j4

How can I achieve same thing for pip?

Jihun
  • 1,415
  • 1
  • 12
  • 16
  • possible duplicate of [Parallel Pip install](http://stackoverflow.com/questions/11021130/parallel-pip-install) – Chris Martin Oct 07 '14 at 07:02
  • 5
    I don't think this is duplicated question. In "Parallel Pip install" thread, if I understood correctly, the author wants to run multiple pip install processes. In contrast, what I want to do is install a package using multiple cores. – Jihun Oct 07 '14 at 07:10
  • 1
    as far as I know this is not implemented. The command build_ext has to be customized to achieve this. – marscher Oct 23 '14 at 18:47
  • 2
    Silly question; would it be possible to do export MAKEFLAGS=-j5 prior to running pip for pip packages that build C pieces or does pip not (generally) call out to make for those cases (I haven't dug deep enough to know) – Foon Oct 30 '14 at 14:48

3 Answers3

52

The Ultimate Way to Resolve This Problem

Because all the c / cpp files would be compiled by using make commend, and make has an option which specify how many cpu cores shoule be used to compile the source code, we could do some tricks on make.

  1. Backup your original make command:

    sudo cp /usr/bin/make /usr/bin/make.bak

  2. write a "fake" make command, which will append --jobs=6 to its parameter list and pass them to the original make command make.bak:

    make.bak --jobs=6 $@

So after that, not even compile python with c libs, but also others contain c libs would speed up on compilation by 6 cores. Actually all files compiled by using make command will speed up.

And good luck.

Use: --install-option="--jobs=6" (pip docs).

pip3 install --install-option="--jobs=6" PyXXX

I have the same demand that use pip install to speed the compile progress. My target pkg is PySide. At first I use pip3 install pyside, it takes me nearly 30 minutes (AMD 1055T 6-cores, 10G RAM), only one core take 100% load.

There are no clues in pip3 --help, but I found lots of options like pip install -u pyXXX, but I didn't know what is '-u' and this parameter was not in pip --help too. I tried 'pip3 install --help' and there came the answer: --install-option.

I read the code of PySide's code and found another clue: OPTION_JOBS = has_option('jobs'), I put ipdb.set_trace() there and finally understand how to use multicore to compile by using pip install.

it took me about 6 minutes.

--------------------------update------------------------------

as comment below, I finally used tricks like this: cd /usr/bin sudo mv make make.bak touch make then edit make: vim make or other way you like and type this: make.bak --jobs=6 $* I'm not familiar with bash, so I'm not sure if this is the correcct bash code. I'm writing this comment in windows. The key is rename make into make.bak, and then create a new make, use this new make to call make.bak with added param --jobs=6

Kevin
  • 2,234
  • 2
  • 21
  • 26
Plasmatium
  • 620
  • 6
  • 7
  • Seems to work -- the processor load increases on all of my cores when I run it this way. – BenB Feb 09 '16 at 20:11
  • 4
    I get warning `UserWarning: Disabling all use of wheels due to the use of --build-options / --global-options / --install-options.`. It makes sense, but is it likely that the performance benefits of using multiple cores outweighs the performance hit of not using wheels? What does it depend on? – max Sep 21 '16 at 22:56
  • 2
    What about for pip (not pip3)? I'm getting: `error: option --jobs not recognized` – timotheecour Sep 23 '16 at 20:13
  • Have tried pip2.7 am getting same error error: option --jobs not recognized it's for package specific ? – Dhanasekaran Anbalagan Oct 24 '16 at 16:33
  • 30
    It appears that the parameter passed to `--install-option` is simply forwarded to the `setup.py` for the package you're installing. As such, the `--jobs=6` **is specific to PySide's build process**. Other packages may have similar (or the same) build flag, but it's not a generic option. – Fake Name Feb 22 '17 at 00:19
  • Sorry for a long time passed. I finally used tricks as below: `cd /usr/bin` `sudo mv make make.bak` `touch make` then edit make: `vim make` or other way you like and type this: `make.bak --jobs=6 $*` I'm not familiar with bash, so I'm not sure if this is the correcct bash code. I'm writing this comment in windows. The key is rename make into make.bak, and then create a new make, use this new make to call make.bak with added param --jobs=6 – Plasmatium Apr 12 '17 at 07:50
  • This answer seems to apply to only PySide. It doesn't work for any old Python 2.7 package. setuptools for Python 2.7 does not use make. – nmgeek May 02 '18 at 19:16
  • 1
    There's no need to move system files or fool around with /usr/bin at all. Just add the script to /usr/local/bin/ (or ~/.local/bin/) and reference /usr/bin/make instead of /usr/bin/make.bak. I believe this is much cleaner and won't get nuked by system updates. – Adam Gaskins May 10 '20 at 16:18
  • The actual command in the replacement `make` script would be better if you run `$0.bak` as the command rather than explicitly `make.bak`; that'll substitute the actual command it's being invoked as and make it a drop-in replacement for make even on a system where it's installed in a different directory, or invoked directly via the file path rather than a lookup into `$PATH`. (With the appropriate quoting/escaping the correct command line would be `"$0.bak" --jobs=6 "$@"`.) – AJMansfield May 18 '23 at 10:59
32

Tested this works https://stackoverflow.com/a/57014278/6147756

Single command:

MAKEFLAGS="-j$(nproc)" pip install xxx

Enable for all commands in a script:

export MAKEFLAGS="-j$(nproc)"
Mouagip
  • 5,139
  • 3
  • 37
  • 52
Yushu
  • 429
  • 5
  • 6
  • 2
    Hi, Yushu. Please link to the answer you are referring to instead of the question. You can click “share” under an answer to obtain a direct link. – fpersyn Aug 23 '21 at 14:32
  • This is by far the simplest answer if you're using a script to call pip with. – Kevin Meyer Mar 21 '22 at 00:52
  • This is a much better solution than the accepted solution. I would advise against replacing the system-wide `make` with a script that includes the argument `--jobs=6`. At the very least, this hack will stop working if a new version of make is included in an update. – mallwright Mar 15 '23 at 13:06
  • As of 2023 with Python 3.10, this is what worked! – legel Mar 21 '23 at 18:36
8

From what I can tell it does not look like pip has this ability but I may be mistaken.

To do multiprocessing in python you use the multiprocessing package, [here is a guide I found] (http://pymotw.com/2/multiprocessing/basics.html) about how to do it if you are interested and this is a link to the python docs that talk about it. I also found this question useful, Multiprocessing vs Threading Python, to make sure that multiprocessing did what I thought it did, being take advantage of multiple CPUs.

I have gone through the pip source code (available here) looking for a reference to the multiprocessing package and did not find any use of the package. This would mean that pip does not use/support multiprocessing. From what I can tell the /pip/commands/install.py file is the one of interest for your question as it is called when you run pip install <package>. For this file specifically the imports are 

from __future__ import absolute_import

import logging
import os
import tempfile
import shutil
import warnings

from pip.req import InstallRequirement, RequirementSet, parse_requirements
from pip.locations import virtualenv_no_global, distutils_scheme
from pip.basecommand import Command
from pip.index import PackageFinder
from pip.exceptions import (
    InstallationError, CommandError, PreviousBuildDirError,
)
from pip import cmdoptions
from pip.utils.deprecation import RemovedInPip7Warning, RemovedInPip8Warning

which you can see does not have any reference to the multiprocessing package but I did check all of the other files just to be sure.

Furthermore, I checked the pip install documentation and found no reference to installing using multiple cores.

TL;DR: Pip doesn't do what you are asking. I may be wrong as I didn't look at the source that long but I'm pretty sure it just doesn't support it.

Community
  • 1
  • 1
guribe94
  • 1,551
  • 3
  • 15
  • 29