17

I create a python3 virtual environment (explicitly avoiding symlinks, with --copies):

» python3 -m venv --without-pip --copies venv

This is my complete virtual environment now:

» tree venv/
venv/
├── bin
│   ├── activate
│   ├── activate.csh
│   ├── activate.fish
│   ├── python
│   └── python3
├── include
├── lib
│   └── python3.4
│       └── site-packages
├── lib64 -> lib
└── pyvenv.cfg

I disable the PYTHONPATH, to make sure nothing is leaking from outside:

» PYTHONPATH=""

Activate the venv:

» source venv/bin/activate

Verify that activate has not polluted my PYTHONPATH:

» echo $PYTHONPATH

(blank, as expected)

I am using the right python:

» which python
/foo/bar/venv/bin/python

But the system modules are still being accessed:

» python 
Python 3.4.3 (default, Oct 14 2015, 20:28:29) 
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import unittest
>>> print(unittest)
<module 'unittest' from '/usr/lib/python3.4/unittest/__init__.py'>
>>>

I would expect the import unittest statement to fail, since the virtual environment has no such module.

I would like to know:

  • Why are system packages accessed when in a virtualenv?
  • How can I create a completely self-contained virtual environment?
VitalyZ
  • 445
  • 1
  • 4
  • 12
blueFast
  • 41,341
  • 63
  • 198
  • 344

2 Answers2

9

If I recall correctly the core system packages are symlinked, so they are the same files (partly to keep the size of the virtualenv down).

The default is not to include the site-packages directory, so it won't access 3rd party libraries that have been installed.

If you want to truly isolated and self-contained virtual environment, you might be better off looking at docker.

Virtualenv is really more of a lightweight way of managing different 3rd party installed packages for different apps.

EDIT:

It looks like --always-copy doesn't actually always copy all files:

virtualenv doesn't copy all .py files from the lib/python directory

Digging into the source and it looks like there's a smallish set of modules that are deemed to be "required" and these are the ones that are copied:

https://github.com/pypa/virtualenv/blob/ac4ea65b14270caeac56b1e1e64c56928037ebe2/virtualenv.py#L116

Edit 2:

You can see that the old python directories still appear in the sys.path, but after the directories for the virtualenv itself:

>>> import sys
>>> sys.path
['', '/home/john/venv/lib/python2.7', 
'/home/john/venv/lib/python2.7/plat-linux2', 
'/home/john/venv/lib/python2.7/lib-tk',
'/home/john/venv/lib/python2.7/lib-old', 
'/home/john/venv/lib/python2.7/lib-dynload', '/usr/lib/python2.7',
'/usr/lib/python2.7/plat-linux2',
'/usr/lib/python2.7/lib-tk',
'/home/john/venv/local/lib/python2.7/site-packages',
'/home/john/venv/lib/python2.7/site-packages']
Community
  • 1
  • 1
John Montgomery
  • 8,868
  • 4
  • 33
  • 43
  • Please note that I am using `--copies`: no symlinks are used (as seen in the directory tree). Docker is overkill for my needs: I can *live* with this situation, but I want to understand the background issues. Maybe a built-in PYTHONPATH in the system's python binary (which gets copied to the virtualenv)? How is this done? The system `python` binary is installed (in this case) with `apt get`, so how would the python binary know where it is installed (specially after being copied around!)? – blueFast Jan 27 '16 at 11:11
  • Hadn't spotted the `--copies` there. I tested out using `--always-copy` locally (is `--copies` the older name?) and that does copy the files, but strangely *not* for unittest. If you try the same test for the `os` module, you should see that gives you the right path. – John Montgomery Jan 27 '16 at 14:23
  • I do not have the `--always-copy` option, but `--copies`. I do not know which one is older. I am using python 3.4.3, and you? – blueFast Jan 28 '16 at 09:35
  • And thanks for your explanation. It clarifies *why* are certain packages not copied with virtualenv (for me nothing is copied, actually, who knows why), but I still would like to know how can the python in the virtualenv use system packages (like unittest) – blueFast Jan 28 '16 at 09:42
  • I was actually trying this with virtualenv 13.1.2 (on linux), but the docs only seem to refer to `--always-copy`. Odd. – John Montgomery Jan 28 '16 at 11:14
  • The old python is still referenced on the PYTHONPATH - possible hardcoded in a binary. – John Montgomery Jan 28 '16 at 11:19
  • Ok, got it. As you say, probably embedded in the python binary during compilation. As mentioned, my python is installed with apt, so using pre-compiled packages. It seems odd that such "installation-phase" info is hardcoded in a binary, though: it makes the installation directory for the binary not really relocatable (so basically, it can only be correctly installed in the path hardcoded during compilation time) – blueFast Jan 28 '16 at 11:40
  • You can use `--relocatable` to make the virtualenv movable. It's quite common for binaries to have a hardcoded library path though, as it means you can multiple versions installed with different paths encoded in them. Everything then just works as expected. – John Montgomery Jan 28 '16 at 11:42
  • Thanks, I know about `--relocatable`, but that is another matter altogether. What annoys me is that the python binary itself (not the virtualenv), has a hardcoded dependency to an installation path which is **none of its business**. I am the one to install it wherever I see fit (or `apt` in case of system python, or `virtualenv` in case of the virtualenv's python). – blueFast Jan 28 '16 at 11:50
  • I would be less annoyed with python having a hardcoded *relative* reference to a `lib` path, but not with an absolute path. I assume these are linux/unix conventions, and there is surely a rationale of why this is the case (my last *linux from scratch* installation was done some time ago), but it is still surprising. – blueFast Jan 28 '16 at 11:51
0

you can build python from source on an old Linux; then the python will be complete self-contained and able to run on many Linux.

in the reference link, it is the step detail to build python 3.6 on centos 5 in container env. the output can be used in many Linux like debian 10, centos 6, 7, 8, photon 3, alpine, ...

  • readline-devel: enable python command line history
  • sqlite-devel: enable sqlite support
  • expat-devel: enable xmltree support
  • bzip2-devel: enable bzip2 support
  • compile SSL from source to avoid dynamic lib link
  • if higher version of python, maybe you also need to compile higher version of gcc to get sth like full C++11 and higher support
  • after compiling, you need to download setuptools and pip from pypi.python.org and do python setup.py install to get pip ready for this standalone python.

ref: https://github.com/stallpool/track-network-traffic#mitmproxy-for-manylinux

Doz Parp
  • 279
  • 4
  • 23