41

I'm looking for a method to use pip or similiar to install a list of python packages to a custom target directory (ex./mypath/python/pkgs/ ), but also exclude/blacklist specific dependencies.

I want to exclude specific dependencies since they are already met from a different install path (e.g. an anaconda install). I don't have the privilege of adding packages to the default python installation (nor do I want to).

I'm currently use the -r and -t options of pip. But have not found a way to exclude specific packages.

A pip command like this is would be ideal:

pip install --log pip.log -r req.txt -t /mypath/pypkgs/ --exclude exclude.txt

--no-deps is not an option since I need some of the dependencies.

I'm currently pursuing a python script to do pip installs that include dependencies I don't need via:

pip install --log pip.log -r req.txt -t /mypath/python/pkgs/

and then (automatically) remove the unneeded dependencies after the pip install finishes.

I hoping some combination of pip commands can achieve what I'm looking for some straightforward away. I'm using pip 7.1.2. Thanks!

Similar, yet I'm not upgrading and want to specify a target path:

pip: upgrade package without upgrading particular dependency

bscipio
  • 551
  • 1
  • 5
  • 5
  • So you want to use `pip` to install packages for an interpreter other than the one you're using to run it? – jonrsharpe Oct 30 '15 at 16:52
  • I'm not sure I'm answering your question @jonrsharpe: When I use pip from my default install (anaconda) with -t [targetdir] it appears to ignore the fact that I already it (For instance, It will install numpy again into [targetdir]). I want to add extra packages in my target directory without duplicate dependencies for my default install/interpreter. I plan to add the custom path to my PYTHONPATH. – bscipio Oct 30 '15 at 17:26

5 Answers5

29

Faced with a similar problem, and using a running bash shell I managed to exclude specific packages with

pip install $(grep -ivE "pkg1|pkg2|pkg3" requirements.txt)

where pkg1 and so on are the names of the packages to exclude.

kodemartin
  • 440
  • 6
  • 9
  • 3
    In case you made some comments then you would have to remove them as well: `pip install $(grep -ivE "^pkg1|^pkg2|^#|^\s*$" requirements.txt | sed s/#.*//g)` – Night Train Apr 23 '20 at 12:03
  • 1
    But if your requirements.txt also includes specific version requirements it doesn't work. in you requirements.txt it will look like `pip == 20.0.2` but when calling it with the above command it should look like `pip install 'pip==20.0.2'`. Any idea how to solve that problem? – Night Train Apr 23 '20 at 12:05
  • 2
    I just found out this one works just fine! `pip install -r <(sed '/^[pk1|pkg2]/d' requirements.txt)` – Night Train Apr 23 '20 at 12:37
  • This also doesn't work if requirements.txt includes another file ("-r morerequirements.txt"). – Seppo Enarvi Jul 24 '22 at 10:07
17

I think this essentially can be achieved in several steps, assuming you're using virtualenv or similar...

  • If you first do a normal pip freeze > requirements.txt you'll get all of the transitive dependencies (e.g. not excluding anything.)

  • Now edit requirements.txt, remove the packages you want to exclude...

  • Finally, in a new environment do pip install -r requirements.txt -t ... --no-deps. Voila: you've installed the dependencies you wanted while excluding specific ones.

thom_nic
  • 7,809
  • 6
  • 42
  • 43
  • 1
    Is it possible to share virtualenv with other users? My use case involves pointing colleagues to a path they can be appended to their PYTHONPATH environment variable, and we're done. The goal being a standard/default python environment we can all share (containing custom and 3rd party modules). This custom path will build upon a vanilla anaconda installation. (OS: Linux). – bscipio Aug 03 '16 at 22:18
  • You mean like a network share with pre-installed binaries? That smells like a bad idea but I suppose it would work as long as everyone has access to the same path. What's wrong with doing `pip install`? – thom_nic Aug 09 '16 at 20:36
  • Right. We desire to use a shared network path so we have a standard environment(s) to share among the team - in addition to anaconda. We are in a firewalled environment - so `pip install` isn't an option (We'll use pip outside the firewall, and then transfer the environment inside the firewall so we can all share. That seems easier to manage than each potentially having a different environment). Going back to original question, because of anaconda we want to avoid duplicate dependencies (numpy) , however `pip -t /path/` doesn't consider the current environment. – bscipio Aug 12 '16 at 18:45
7

You can use pip install --no-deps with pip check to exclude specific packages. A real example of mine is when I installed paddleocr on Jetson Nano, pip kept installing python-opencv for me, which has already installed by myself without using pip, but pip cannot detect it. To stop pip installing python-opencv for me, here are the steps

  1. use pip install --no-deps paddleocr to install paddleocr without its dependencies
  2. use pip check to list unresolved dependencies, reformat the output so that it can be read by pip install -r, then remove the packages you don't want to install (it is python-opencv in my case), and save it to a file named fix-deps.txt
  3. use pip install --no-deps -r fix-deps.txt to install unresolved dependencies
  4. repeat steps 2 and 3 until the output of pip check only contains the packages you don't want to install.

pip install --no-deps and pip check are very useful commands that allow you to resolve the dependencies by yourself when pip cannot do it right for you. The shortcoming of this solution is the output of pip check is designed for humans, it cannot be used by pip install -r directly, so you have to reformat the output manually or use the awk command.

PR and issue have been created for the pip community to make the output of pip check suitable for the pip install to read, but for some reason, they don't think it is a good idea. so I guess this is the best we can do now.

Refs:

  1. https://github.com/pypa/pip/pull/10108
  2. https://github.com/pypa/pip/issues/10066#issuecomment-872638361
link89
  • 1,064
  • 10
  • 13
4

An approach that takes into account sub-dependencies is to first install the exclude.txt environment, then the req.txt environment, then check the diff and finally install that into the target directory.

Example using virtualenv that will work on GNU/Linux:

virtualenv tmpenv
source tmpenv/bin/activate

pip install -r exclude.txt
pip freeze > exclude-with-deps.txt
pip install -r req.txt
pip freeze > req-with-deps.txt
comm -13 exclude-with-deps.txt req-with-deps.txt > final-req.txt

pip install -r final-req.txt --no-deps -t pypkgs
edvardlindelof
  • 206
  • 2
  • 4
0

Quick Answer:
pip freeze | grep -vFxf pip_ignore.txt > requirements.txt
Will ignore whatever you save in pip_ignore.txt

This is good if you already know what you're going to ignore. However if you're about to install something that you know you'll want to ignore, you're going to want to be able to handle things a bit more easily. In that scenario what I do is pip freeze > before.txt then I install whatever packages it is, and then pip freeze | grep -vFxf before.txt > new_stuff_to_ignore.txt. You now have a list of the stuff you just installed. You can add it to your pip_ignore.txt and then finally do pip freeze | grep -vFxf pip_ignore.txt > requirements.txt

Be mindfudl that some packages in your requirements may share dependencies with packages you want to ignore. These dependencies might end up in your pip_ignore.txt.

David Mendes
  • 197
  • 2
  • 4