pip requirement.txt conditionals or environment markers based on env variables

Question

Is there a way to specify conditional installs in a pip requirements.txt file that are based on the value of an environment variable?

I've been able to control most of what I need with environment markers, but all the markers I know of work only with specific values that are essentially pre-defined by python.

For example, I want to be able to control package installation for RHEL 5.3 vs. RHEL 6.3 vs. RHEL 6.6, etc. Also based on other criteria.

It would be perfect if I could specify in the results.txt file an environment variable that would be checked against a value I set before running pip. This seems like it would be desirable and straight forward functionality. I haven't had much luck so far finding comprehensive discussions of environment markers, so I'm hoping I've just missed some key bit of information :-)

Thanks much.

score 7 · Answer 1 · answered Feb 12 '16 at 02:01

There's not really a way to do it with environment variables. Pip requirements files are basically just a list of pip install arguments listed in a file. So if your requirements file looks like this:

Foo==1.1.0
Bar==0.1.0
Baz==2.0.1

Logically, pip is doing this:

pip install Foo==1.1.0
pip install Bar==0.1.0
pip install Baz==2.0.1

Unfortunately, in that context there's no way to apply environment variables.

There are a couple solutions to this problem.

One, you could have a base requirements file, say requirements.txt, that lists common dependencies for all platforms. Then, you could have individual requirements files that are named, say, requirements.rhel53.txt, requirements.rhel63.txt, etc. The top of each of these files could have this as the first line:

-r requirements.txt

And then additional special dependencies listing. Then, in each environment, you could set an env var, let's call it $PLATFORM, and run a command like this to install dependencies:

$ pip install -r requirements.$PLATFORM.txt

Or, you could use constraints files. Your requirements.txt would just list dependencies without versions:

Foo
Bar
Baz

And then you could have a constraints file, again for each platform, that would list specific version requirements. For example, you could have constraints.rhel53.txt that had this:

Foo==1.1.0
Bar==0.1.0
Baz==2.0.1

And again, you set an env var and then run a command like this:

$ pip install -r requirements.txt -c constraints.$PLATFORM.txt

It's a cumbersome solution, but that would be one way of dealing with it. Unfortunately, pip does not have a native solution.

Thanks for the feedback mipadi. I was afraid of that. Since, as I understand it, the pip environment markers are really just equivalents to associate functionality in python (like `platform_system` is really `platform.system()` in python,) I was hoping there was a way to have a marker equivalent to something like `os.environ.get('MYVARIABLE')`. The pain I have is that there are multiple dimensions to my requirement criteria, so using multiple requirement files gets a bit messy to maintain. Using the constraints files is a good idea. I'll try to at least reduce the complexity that way. — ghoff, Feb 12 '16 at 18:43
Using different constraint files could work around my problem fairly smoothly **except** that I can't find a way to make a constraint file completely disallow a package. Is there a way? I tried working around this by specifying a non existent version (e.g. `six <= 0.1`) but then pip errors because it can't find an acceptable version to install. Is there a way to tell pip not to error out, or to warn rather than error? — ghoff, Feb 13 '16 at 00:37
I also tried using an environment marker that evaluates to false (e.g. `six; sys_platform == 'win32'` in a linux environment). But instead of constraining the installation of the package, pip constrains (ignores) this line of the constraint file and installs the latest version of the package (because it's unconstrained in the requirements.txt file.) — ghoff, Feb 13 '16 at 00:40

score 1 · Answer 2 · edited May 23 '17 at 10:34

I ended up with a scripted solution similar to one I found in: Is there a way to have a conditional requirements.txt file for my Python application based on platform?

Basically, I did something like the following (partial) example, which allows me to define packages as needed in the script, and still pass in requirements.txt and constraints.txt files. I'm new to Python, so please forgive (or comment on) any less-than-optimal coding practices:

import pip, sys, getopt

def install(options, packages):
    pipargs = options + packages
    pip.main(pipargs)

constraints_file = ''
requirements_file = ''

try:
    opts, args = getopt.getopt(sys.argv[1:],"hr:c:",["help","requirements-file=","constraints-file="])
except getopt.GetoptError:
    print_usage()
    sys.exit(2)
for opt, arg in opts:
    if opt in ("-h", "--help"):
       print_usage()
       sys.exit()
    elif opt in ("-r", "--requirements-file"):
        requirements_file = arg
    elif opt in ("-c", "--constraints-file"):
        constraints_file = arg

base_pkgs = [
    "nose>=1.3.4",
    "wheel",
    "pytz"
]

foo_pkgs = [
    "Pygments; python_version == '2.7'",
    "rednose; python_version == '3.5'",
]

bar_pkgs = [
    "six",
]

if __name__ == '__main__':

    from os import environ as env

    myvar = env.get('MYVAR')
    if(myvar == 'foo'):
        mypkgs = foo_pkgs
    elif(myvar == 'bar'):
        mypkgs = bar_pkgs
    else:
        mypkgs = ''

    pkglist = base_pkgs + mypkgs

    pipoptions = [  'install',
                    '--upgrade',
                    '--force-reinstall'
                 ]

    if(constraints_file):
        pipoptions += ['-c', constraints_file]
    if(requirements_file):
        pipoptions += ['-r', requirements_file]

    install(pipoptions, pkglist)

This was extreemely helpful for me. Was not aware packages could be installed using pip within a python script. neat! — Bobs Burgers, Aug 30 '20 at 03:50

pip requirement.txt conditionals or environment markers based on env variables

2 Answers2

Linked

Related