66

I know how to install packages in Anaconda using conda install and also how to install packages that are on PyPi which is described in the manual.

But how can I permanently include packages/folders into the PYTHONPATH of an Anaconda environment so that code that I am currently working on can be imported and is still available after a reboot?

My current approach is to use sys:

import sys
sys.path.append(r'/path/to/my/package')

which is not really convenient.

Any hints?

starball
  • 20,030
  • 7
  • 43
  • 238
Cord Kaldemeyer
  • 6,405
  • 8
  • 51
  • 81
  • I don't understand what *"permanently include packages/folders in Anaconda e.g. with code that I am currently working on"* means, do you mean a) `conda and pip` are not locally installing them into the right conda environment (in which case do `conda activate/deactivate` until you get to the right Python environment, you may have several in conda), or b) you want to manually bundle third-party packages with your package(? sounds like a terrible idea, just define package dependencies). What does "permanently" mean, on which machine? – smci Dec 16 '18 at 03:46
  • 1
    I just want to be code that I am working on to be available/importable within a specific environment even after reboot. So I have specified the question above. Actually, I found different solutions for my problem which I have provided below. But let me know if there are others ways and I'll add them! – Cord Kaldemeyer Dec 17 '18 at 07:26

4 Answers4

115

I found two answers to my question in the Anaconda forum:

1.) Put the modules into into site-packages, i.e. the directory $HOME/path/to/anaconda/lib/pythonX.X/site-packages which is always on sys.path. This should also work by creating a symbolic link.

2.) Add a .pth file to the directory $HOME/path/to/anaconda/lib/pythonX.X/site-packages. This can be named anything (it just must end with .pth). A .pth file is just a newline-separated listing of the full path-names of directories that will be added to your path on Python startup.

Alternatively, if you only want to link to a particular conda environment then add the .pth file to ~/anaconda3/envs/{NAME_OF_ENVIRONMENT}/lib/pythonX.X/site-packages/

Both work straightforward and I went for the second option as it is more flexible.

*** UPDATE:

3.) Use conda develop i. e. conda-develop /path/to/module/ to add the module which creates a .pth file as described under option 2.).

4.) Create a setup.py in the folder of your package and install it using pip install -e /path/to/package which is the cleanest option from my point of view because you can also see all installations using pip list. Note that the option -e allows to edit the package code. See here for more information.

Thanks anyway!

Cord Kaldemeyer
  • 6,405
  • 8
  • 51
  • 81
  • 2
    Still waiting for [`conda develop`](https://conda.io/docs/commands/build/conda-develop.html) :/ – Austin A Jun 29 '18 at 12:32
  • 3
    [`conda develop`](https://conda.io/docs/commands/build/conda-develop.html), basically a script that achieves option 2, seems to be functional now. – Austin A Oct 30 '18 at 13:41
  • 1
    #2 doesn't seem to work for me (I don't seem to have the same directory structure as well; I don't have multiple options for python.X.X and my 'lib' is upper case) – information_interchange Nov 06 '18 at 00:16
  • 1
    @information_interchange Quite plausibly the cause for these differences in directory structure are caused by `Windows vs. Linux` implementation. Under Windows, the places where python looks as it builds `sys.path` are slightly different. See https://docs.python.org/3/library/site.html I believe the path in Windows would be `....path_to_anaconda/lib/site-packages` (also the upper/lowercase would be irrelevant in Windows) – mjv Jul 24 '19 at 14:57
  • I wonder what happen with the pip install option if I change the code afterwards. Need the package to be reinstalled? Can this solution give problems? pythonpath mechanism was an easy way to include code under development, I cannot say what is the best alternative. – Vincenzooo Jan 15 '20 at 21:42
  • If you install it as "editable mode" via `pip install -e package_name` you can change the code within package directly. See: https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs – Cord Kaldemeyer Jan 16 '20 at 07:55
  • Option 4 worked for me. It is much easier to add to your path from linux, but if you are stuck on Windows, this is a good solution. – mikey Jul 30 '20 at 18:06
  • FWIW, I haven't been able to get the `conda develop` approach to work. According to [this link](https://github.com/conda/conda-build/issues/1992) it doesn't seem like it really works at this point. – Brett Aug 19 '20 at 16:29
  • `conda develop` can accept multiple path at once, so if your path contais spaces and you pass it to `conda develop` without enclosing it in quotes, several lines (with wrong paths) are added to the .pth file. If you pass it enclosed in quotes, a single line is added to .pth, but of course it contains spaces, I didn't understand yet if this works with anaconda or not (it doesn't for me, but I am sure in my case there are other problems). – Vincenzooo Mar 22 '22 at 08:50
  • 2
    make sure you run `conda install conda-build` first – user108569 Jul 19 '22 at 18:10
  • When I add `my.pth` file to `/opt/conda/lib/python3.10/site-packages/`, the path shows up in the base environment, but it doesn't show up in another environment. Is this intended or am I missing something? – Prateek Verma May 25 '23 at 02:42
13

I'm able to include local modules using the following: 

conda-develop /path/to/module/

I hope it helps.

Ariel Werle
  • 131
  • 1
  • 3
  • Thanks for the hint! I have adapted my answer as the command only creates a `.pth` file as described under 2.). – Cord Kaldemeyer Mar 18 '19 at 08:13
  • 1
    **Tip** (got hung up on this): /path/to/module shouldn't include the module name.; just the parent directory's path. – Joshua R. Jul 03 '19 at 07:30
  • Note: If there is some element of the package that has to be compiled, such as a Cython or f2py code, you can use: python setup.py build_ext --inplace – Ariel Werle Jul 05 '19 at 16:11
  • @ArielWerle, can you delete the path in sys.path permanently (https://stackoverflow.com/questions/63011104/how-to-delete-a-certain-path-in-sys-path-in-anaconda-env-base) ? – Joonho Park Jul 21 '20 at 09:20
  • 1
    "'conda-develop' is not recognized as an internal or external command" – anishtain4 Jan 20 '21 at 20:59
4

The way I do this, which I believe is the most native to conda, is by creating env_vars.sh files in my environment, as per the official documentation here.

For macOS and Linux users, the steps are as follows:

  1. Go to your environment folder (e.g. /miniconda1/env/env_name). $CONDA_PREFIX is the environemnt variable for your environment path.

    cd $CONDA_PREFIX

  2. Create the activate.d and deactivate.d directories.

    mkdir -p ./etc/conda/activate.d
mkdir -p ./etc/conda/deactivate.d

  3. Inside the each respective directory, create one env_vars.sh file. The one in the activate.d directory will set (or export) your environment variables when you conda activate your environment. The file in the deactivate.d directory will serve to unset the environment variables when you conda deactivate your environment.

    touch ./etc/conda/activate.d/env_vars.sh
touch ./etc/conda/deactivate.d/env_vars.sh

  4. First edit the $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh to export the desired environment variables.

    #!/bin/sh

export VAR_A='some-thing-here'
export VAR_B=/path/to/my/file/

  5. Afterwards, open to edit the $CONDA_PREFIX/etc/conda/deactivate/env_vars.sh, in order to unset the env variables when you conda deactivate like so:

    #!/bin/sh

unset VAR_A
unset VAR_B

Again, the source of my description comes straight from the conda docs here.

Newskooler
  • 3,973
  • 7
  • 46
  • 84
1

Just to add to Cord Kaldemeyer's answer above, for the 2nd option. If you only want to link to a particular conda environment then add the .pth file to ~/anaconda3/envs/{NAME_OF_ENVIRONMENT}/lib/pythonX.X/site-packages/

Vik
  • 690
  • 7
  • 22