Create a virtualenv from another virtualenv

Question

Can we create a virtualenv from an existing virtualenv in order to inherit the installed libraries?

In detail:

I first create a "reference" virtualenv, and add libraries (with versions fixed):

virtualenv ref
source ref/bin/activate
pip install -U pip==8.1.1     # <- I want to fix the version number
pip install -U wheel==0.29.0  # <- I want to fix the version number

Then:

virtualenv -p ref/bin/python myapp
source myapp/bin/activate
pip list

I get:

pip (1.4.1)
setuptools (0.9.8)
wsgiref (0.1.2)

How to get my installed libraries?

Similar question

I saw a similar question: Can a virtualenv inherit from another?.

But I want a isolated virtualenv which didn't use the referenced virtualenv, except for libraries installation. So, adding the specified directories to the Python path for the currently-active virtualenv, is not the solution.

Why doing that?

Well, we have an integration server which builds the applications (for releases and continuous integration) and we want to keep the control on libraries versions and make the build faster.

Create a relocatable virtualenv

I think I could use a relocatable virtualenv, that way:

create the ref virtualenv
make it relocatable: ``virtualenv --relocatable ref```

For "myapp":

copy ref to myapp

What do you think of this solution? Is it reliable for a distribuable release?

Create a second virtualenv and then `pip install -r <(/path/to/other/pip freeze)`? — Phillip, Sep 09 '16 at 10:26
You're right, I can install the requirements again. But if we can only copy the libraries, it can go faster. — Laurent LAPORTE, Sep 09 '16 at 10:31
What do you think of the "Create a relocatable virtualenv" solution? — Laurent LAPORTE, Sep 09 '16 at 11:34

score 10 · Answer 1 · answered Jan 17 '18 at 17:27

10

You can solve your problem by using .pth files. Basically you do this:

virtualenv -p ref/bin/python myapp
realpath ref/lib/python3.6/site-packages > myapp/lib/python3.6/site-packages/base_venv.pth

After doing this and activating myapp, if you run pip list you should see all the packages from ref as well. Note that any packages installed in myapp would hide the respective package from ref.

answered Jan 17 '18 at 17:27

dcmm88

1,445
1
12
30

1

Works with environments created using the built-in `venv` module, too. Thanks for the tip! – ayanami Mar 26 '18 at 14:32
With Python 3.6 (from the rh-python36 SCL for CentOS 7), this works as described, but even though `pip list` shows packages from `ref`, if some software not under my control tries to use `pip install` to install one of the packages that is in `ref`, instead of saying it's already installed, it actually tries to install it. Using `--exists-action` doesn't seem to help. I suppose this may be useful behaviour in some cases, but not mine! – doshea Aug 05 '20 at 11:18
I think the issue I described with `pip install` trying to install packages that are already in `ref` is due to https://github.com/pypa/pip/issues/8695 pip is bundled in virtualenv, and virtualenv 20.0.30 includes a version of pip that suffers from that bug. – doshea Aug 06 '20 at 01:51

score 3 · Answer 2 · answered Sep 09 '16 at 11:15

3

You may freeze list of packages from one env:

(ref) user@host:~/dir$ pip freeze > ref-packages.txt

Then install them:

(use) user@host:~/dir$ pip install -r ref-packages.txt

answered Sep 09 '16 at 11:15

Sergey Belash

1,433
3
16
21

No, I don’t want to re-install the libraries, just copy existing ones (if possible). – Laurent LAPORTE Sep 09 '16 at 11:24
1

So, what's the difference between "re-install" and "just copy"? I this case pip won't download them, just unpack the same libraries to an other directory. – Sergey Belash Sep 09 '16 at 11:30
Well, libraries with binaries (like lxml) run some build during installation… – Laurent LAPORTE Sep 09 '16 at 11:33

score 0 · Answer 3 · answered Sep 09 '16 at 10:28

0

when you install the second virtualenv you have to add --system-site-packages flag.

virtualenv -p ref/bin/python myapp --system-site-packages

answered Sep 09 '16 at 10:28

Francesco Nazzaro

2,688
11
20

2

The ``--system-site-packages`` installs the libraries of the system Python, not the ones of the referenced virtualenv. – Laurent LAPORTE Sep 09 '16 at 10:29
1

but the "system" should be your virtualenv – Francesco Nazzaro Sep 09 '16 at 10:34
I’ve done this test too but it fails. I’ll retry that on my build server. I hope ``virtualenv`` is not too old… – Laurent LAPORTE Sep 09 '16 at 11:13
1

Tested doesn't work for me either. pip version 8.1.2 – Sardorbek Imomaliev Sep 09 '16 at 11:15
1

I agree, it doesn’t work. I've tested on OS X. => it copies the system site-package, not the libraries of my newly created virtualenv. – Laurent LAPORTE Sep 09 '16 at 11:23
1

I'm sad to find this doesn't work anymore. It seems like such a great solution for the problem, but I concur, not working in virtualenv 15.1.0, pip 10.0.1 either. – Engineero May 30 '18 at 19:59

Klaus D. · Answer 4 · 2016-09-09T10:42:24.630

0

The pip version 1.4.1 was bundle with an old version of virtualenv. For example the one shipped with Ubuntu 14.04. You should remove that from your system and install the most recent version of virtualenv.

pip install virtualenv

This might require root permissions (sudo).

Then upgrade pip inside the virtual env pip install -U pip or recrete the env.

edited Sep 09 '16 at 10:42

answered Sep 09 '16 at 10:35

Klaus D.

13,874
5
41
48

score 0 · Answer 5 · edited May 23 '17 at 11:46

I think your problem can be solved differently. With use of PYTHONPATH. First we create ref virtaulenv and install all needed packages here

$ virtualenv ref
$ source ref/bin/activate
$ pip install pep8
$ pip list
> pep8 (1.7.0)
> pip (8.1.2)
> setuptools (26.1.1)
> wheel (0.29.0)

Then we create second virtaulenv use.

$ virtualenv use
$ source use/bin/activate
$ pip list
> pip (8.1.2)
> setuptools (26.1.1)
> wheel (0.29.0)

And now we can set our PYTHONPATH in this env to include ref's directories

$ export PYTHONPATH=PYTHONPATH:/home/path_to/ref/lib/python2.7/site-packages:/home/path_to/ref/local/lib/python2.7/site-packages
$ pip list
> pep8 (1.7.0)
> pip (8.1.2)
> setuptools (26.1.1)
> wheel (0.29.0)

As you see this way you just reference installed packages in ref's environment. Also note that we add this folders at the end so they will have lower priority.

NOTE: this are not all folders that exists in PYTHONPATH. I included this 2 because they are main ones. But if you will have some problems you can add other ones too, just lookup needed paths with this method how to print contents of PYTHONPATH

Create a virtualenv from another virtualenv

5 Answers5