1

I have a very big requirements.txt file with many many many dependencies... I have to download and install all of them, so the worst part is the amount. My bandwidth is good, but pip downloads files very slowly, I think there is a limit and I can bypass it using multiple download at once. So is there a way to download multiple dependencies without the order?

If the limit is cached for each ip, I need a solution by bypassing it with shadow proxies:)

apaderno
  • 28,547
  • 16
  • 75
  • 90
75f13af642fc90a7fdca021e6983d2
  • 13
  • 2
  • Refer to this https://stackoverflow.com/questions/9956741/how-to-install-multiple-python-packages-at-once-using-pip – jae Johnson Feb 27 '21 at 11:07
  • No, this is not the solution. – 75f13af642fc90a7fdca021e6983d2 Feb 27 '21 at 11:18
  • 3
    `pip` does currently not support parallel downloads, since it's a bit more complicated than what it might appear as: https://github.com/pypa/pip/issues/825 - also make sure that you're actually spending most of your time downloading - and not compiling (which usually is the case). Usually you'd cache these downloads anyway, so the install only happens once-ish. I'd also think that pinning the version requirement also could save some time, since that could potentially eliminate looking up the most recent version. – MatsLindh Feb 27 '21 at 12:05

1 Answers1

0

pip currently doesn't support parallel downloads https://github.com/pypa/pip/issues/825

But you can speed up the installation.

https://linux.die.net/man/1/xargs

xargs -n 1 -P 4 pip install < requirements.txt

Run up to P processes at a time; the default is 1. If P is 0, xargs will run as many processes as possible at a time.

Ref: https://linux.die.net/man/1/xargs

Zabir Al Nazi
  • 10,298
  • 4
  • 33
  • 60
  • 2
    Wouldn't this invoke `pip` with the same requirements file multiple times, each time starting to resolve the dependencies in the same sequence and possibly just downloading each dependency multiple times? – MatsLindh Feb 27 '21 at 12:03
  • 1
    Instead, a slightly better solution (but which would still potentially download dependencies multiple times) is to slice the requirements file into four different slices, then run those installs in parallel. – MatsLindh Feb 27 '21 at 12:10
  • 1
    no, the syntax is correct. it'll send one row of requirements.txt to pip at a time with four processes happening at once. i didn't know xargs could do this - cool! my question is if pip can be run in parallel. i was under the impression that it needed to be run serially. – keithpjolley Feb 27 '21 at 12:32