1

So, I create a conda env the usual way, but with a slight difference. I copy some extra folders to the env root so I can have them packed.

After packing to tar.gz they are there, but when unpacking and activating, it does not point to the right python executable. any ideas?

╭─ ~/Downloads  
╰─ mkdir -p my_env
╭─ ~/Downloads  
╰─ tar -xzf my_env.tar.gz -C my_env
╭─ ~/Downloads  
╰─ source my_env/bin/activate
╭─ ~/Downloads                                     my_env
╰─ which python
/usr/bin/python
SamuelNLP
  • 4,038
  • 9
  • 59
  • 102

1 Answers1

2

There is an undocumented (probably on purpose, to avoid some user issues) --copy switch that allow you to have a "portable" conda env:

Example:

conda create -p /path/to/new-conda-env --copy python=3 pandas scikit-learn

The conda env created at this path can then be zipped and unzipped elsewhere on similar architecture (ie. linux x64 to linux x64). You can also manually add packages if necessary (copy to site-packages path).

I use this all the time with PySpark + Hadoop to ship all dependencies without having to install them on cluster.

Gaarv
  • 814
  • 8
  • 15
  • so if I have some arbitrary file test.txt I would put it in the env folder prior to the env creation? – SamuelNLP Oct 07 '21 at 08:42
  • If files are just data files and not modules / code you can put them in the env folder after env creation and zip the whole folder and data files together. Your code can then use abolute path or env relative path to find data files when running. – Gaarv Oct 07 '21 at 08:46
  • So this means I'll just compress the env folder instead of using conda pack right? – SamuelNLP Oct 07 '21 at 08:47
  • Yes, exactly. It allows for much easier transition to a machine where conda is not / cant be installed. – Gaarv Oct 07 '21 at 08:50
  • awesome. I'll try this. thank you very much. – SamuelNLP Oct 07 '21 at 08:53
  • another thing, without packing the env there is no bin/activate. How do you activate the env in a machine with no conda – SamuelNLP Oct 07 '21 at 09:21
  • You can juste call the python binary in the env, if you need activate, it's located on a subdirectory below: Example: `source ./lib/python3.9/venv/scripts/common/activate && bin/python3 test.py cat test.py import pandas as pd source ./lib/python3.9/venv/scripts/common/activate && bin/python3 Python 3.9.7 (default, Sep 16 2021, 13:09:58) [GCC 7.5.0] :: Anaconda, Inc. on linux Type "help", "copyright", "credits" or "license" for more information. >>> import pandas as pd >>>` – Gaarv Oct 07 '21 at 09:45
  • BTW sorry, can't find how to format multiline code in comments :/ – Gaarv Oct 07 '21 at 09:51
  • still not working. :/ ```╭─ …/miniconda3/envs   __VENV_DIR__ ╰─ which python /usr/bin/python``` – SamuelNLP Oct 07 '21 at 10:22
  • That's because `which` ask your system. To use this env, you need to run it from its path, ie. `/path/unzipped-env/bin/python3` – Gaarv Oct 07 '21 at 12:07
  • you do not need to call the python executable from the env when it is activated. – SamuelNLP Oct 07 '21 at 14:04
  • Yes but with this method, (--copy) it's not a conda environment, just a Python env which only scope within its directory. It can be used to run scripts / apps using `source /path/to/activate && /path/to/python3 /path/to/app.py` – Gaarv Oct 07 '21 at 14:09
  • Ok, got it. Well, that won't do for me then. :) – SamuelNLP Oct 08 '21 at 09:39
  • At the minimum if you really need conda and can install it, it will work with `conda activate /path/to/env`. You won't need to have to install python + packages. – Gaarv Oct 08 '21 at 11:00