8

While working on some automation scripts, I found the need to have some common packages available to all the scripts (like ipython and spyder) and installing only the, let's call it differential packages for each script. Is there a way to have nested virtual environment in python? That is, some way to set the environments so that python would first looks at the innermost environment; if not found, it would look on the second innermost environment and so on until reaching the system wide environment for a package?

The closest I could find was to use the venv module with the --system-site-packages option but I couldn't achieve what I looking for: first, I don't want to install packages on the system using root permissions; second, I couldn't find a way to nest virtual environments using this option.

What's the best way to achieve this nested virtual environments structure, if there's one?

Guillaume Jacquenot
  • 11,217
  • 6
  • 43
  • 49
user13079354
  • 181
  • 8

3 Answers3

4

You can add a .pth file (a site module feature) to the site packages directory of your derived virtual environment with a line pointing to the site-packages path of your base virtual environment.

In shell, you can do it like this:

# Assumes that the base virtual environment exists, activate it.
. base/bin/activate

# Create the derived virtual environment.
python -m venv ./derived

# Make the derived virtual environment import base's packages too.
base_site_packages="$(python -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])')"
derived_site_packages="$(./derived/bin/python -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])')"
echo "$base_site_packages" > "$derived_site_packages"/_base_packages.pth

base_site_packages is usually base/lib/python<VERSION>/site-packages, the code to get it is taken from https://stackoverflow.com/a/46071447/3063 – same for derived_site_packages.

The packages installed in the base environment will be available in the derived environment. You can verify this by doing pip list in the derived environment.

# Deactivating the base environment is optional,
# meaning that the derived environment can be activated directly too.
deactivate

. ./derived/bin/activate
pip list
palotasb
  • 4,108
  • 3
  • 24
  • 32
3

No, there isn't any such "nesting" feature in venv.

I think your best bet would be to define the common packages in a requirements file and then install those at the time when you create a new virtual environment

pip install -r requirements-common.txt
wim
  • 338,267
  • 99
  • 616
  • 750
  • That's the approach I'm currently using. My complaint about this method is that the total size of my virtual environment is around 3Gb for 23 scripts of less than 1K each. Anyway, if there's no way, that's the answer. – user13079354 Apr 04 '20 at 04:24
  • 1
    If you don't mind a bit of a hack, you could put all the common libs into a directory on disk somewhere and then set the PYTHONPATH environment variable. That should be visible in all venvs. – wim Apr 04 '20 at 07:45
  • 2
    I would use a [`.pth` file](https://docs.python.org/3/library/site.html) rather than modifying `PYTHONPATH`. See this answer and its links for some ideas on how to achieve that: https://stackoverflow.com/a/60974302/11138259 -- in particular this: https://discuss.python.org/t/optimizing-installs-of-many-virtualenvs-by-symlinking-packages/2983/7 – sinoroc Apr 04 '20 at 08:11
  • 1
    @sinoroc Unless I'm missing something obvious, a path file would have to be put in each venv's site. That seems more of a hassle than setting the env var once. Why rather a `.pth` file? – wim Apr 06 '20 at 15:11
  • @wim If the need is to have common libraries available all the time, in all virtual environments and also outside of virtual environments, then yes. I had more of a case-by-case choice in mind. Also, I believe with `.pth` files, it might be possible to achieve a more flexible nesting scenario. But I haven't actually tested all of this. – sinoroc Apr 06 '20 at 15:19
  • 1
    I see. The point of `.pth` files is to allow an installed package to dynamically extend it's *own* import locations at install time. In OP's case would seem like a bit of an abuse of that mechanism, so I wouldn't recommend to use path files here. – wim Apr 06 '20 at 15:30
0

It is possible with nix-shell. I use it just like this - nix-shell <first shell.nix>. Then if needed - in the same session nix-shell <the second shell.nix>

Development environment with nix-shell

So, second session uses underlying first session

Ildar
  • 1
  • 1