83

I have a file mysql.py, which I use in almost all of my projects. Since I do not want to copy and paste the same file into each of these projects I wrote a module - possibly a package in the future.

Question

How do I add a local module to my conda environment, and automatically update or watch it when I change something in mysql.py? How to do the same for a package?

I would like to avoid to set up a local channel/repository and just reference to the folder with mysql.py.

Michael
  • 1,627
  • 1
  • 19
  • 29

3 Answers3

55

If you install the conda build package (and you have a package, not just a script), you can install in "editable" mode:

conda develop .

(running from the directory with your script). This is very similar to the "editable" mode from pip

pip install -e .

Either approach lets you uninstall packages with either

conda develop -u .

or

pip uninstall .

If you just have a script (not a package), you can edit or set the PYTHONPATH environment variable to include the directory with the script.

darthbith
  • 18,484
  • 9
  • 60
  • 76
  • how do you know when to you `pip -e` vs `conda develop`? – Charlie Parker Jan 20 '20 at 19:26
  • 1
    If you're using conda, you should use `conda develop`. The only small problem I ran into with `conda develop` was that it would not install `console_scripts`, whereas `pip install -e` would. This may have been fixed in conda in the meantime though. – darthbith Jan 21 '20 at 13:34
  • how do you check if things were installed? I tried running `conda list` but nothing familiar to my project/packages seem to show up. – Charlie Parker Jan 24 '20 at 20:51
  • 1
    It should show up in the output of `conda list`. Do you have a `setup.py` file properly configured? Does `pip install -e` work? – darthbith Jan 25 '20 at 18:28
  • @CharlieParker See https://jakevdp.github.io/blog/2016/08/25/conda-myths-and-misconceptions/ for pip vs conda. – Michele Piccolini Apr 29 '20 at 12:08
  • After installing conda build, using the conda develop option worked but the package does not show up in conda list. It shows up in pip list and imports fine. I know conda list shows somewhat different items than pip list, but I would expect it to show my package like pip list does. Anyone know why it doesn't? When installed with pip install -e . the package shows up in both pip list and conda list – Mint Oct 13 '20 at 17:03
  • See [http://evantilton.com/guides/anacondapythonpath/](http://evantilton.com/guides/anacondapythonpath/) for a simple method to programmatically change `PYTHONPATH` for a particular conda environment. – E. Davis Jun 26 '21 at 02:02
  • 5
    Note that `conda develop` is not actively maintained and there is a request to deprecate or remove it, see https://github.com/conda/conda-build/issues/4251. So **it is not advisable to use `conda develop` in 2021. – Martin Hepp Nov 12 '21 at 01:32
  • Adding to @MartinHepp: The recommendation seems to remain: use `python setup.py develop` or `pip install -e ` instead, see https://github.com/conda/conda-build/issues/1992. – Justin P. Feb 02 '22 at 14:54
  • @JustinP. that isse at conda is basically a bunch of developers who are super confused about what to do: it seems conda should provide a canonical source of guidance about this. In the meantime, I will try using `pip install -e .` until it doesn't work. – eric Feb 27 '22 at 17:42
  • 1
    Also, conda develop doens't "install" the local module, like by running hte setup, like pip does, is just adds it to the path. – Marc Dec 20 '22 at 17:39
45

While the previous answers are doing what I need, I just want to show what I will be using instead. Since it was my plan to learn about conda packages anyway...

0. Good sources

  1. Michael Sarahan - Making packages and packaging "just work" | YouTube
  2. GitHub - audreyr/cookiecutter: A command-line utility that creates projects from cookiecutters (project templates) and use one of these templates:

1. Create a python package template for conda using cookiecutter

conda install -c conda-forge cookiecutter

Now change to the directory where you want to initialize your package, then do:

cookiecutter https://github.com/conda/cookiecutter-conda-python.git

This will ask for some basic information about the package that you want to create. Then change into your repo

cd myrepo

2. Build your package

make sure conda-build is installed, if not run

conda install conda-build

Make sure to set the CONDA_BLD_PATH as mentioned in anaconda - using a different conda-build root directory - Stack Overflow. This will be the directory where you can find your packages, then run:

conda build conda.recipe

to build your package and clean up after you with

conda build purge

3. Set up your own local channel (no uploading to anaconda.org)

Read

for help.

Index each platform. Maybe someone can confirm that this step is not needed, as my builds already contain the repodata.json. Otherwise:

conda index D:\CODE\condamychannel\win-64

Test if the package can be found with

conda search -c file:///D:\CODE\condamychannel --override-channels mypackage

or add the channel to the config directly (per environment)

conda config --add channels file:///D:\CODE\condamychannel

4. Install (and update) the package

activate myenv

and

conda install mypackage

Once I change mypackage, I give it a new version number in meta.yaml and setup.py and build the package with conda build conda.recipe. Updating is simply

conda update mypackage

See if your package works:

python
>>> import cli from mypackage
>>> cli.cli()
CLI template

This may not be the optimal way, but I could not find a tutorial that contains all the steps I outlined above.

Dogemore
  • 590
  • 1
  • 5
  • 18
Michael
  • 1,627
  • 1
  • 19
  • 29
  • 7
    why did you choose to not use `conda develop` or `pip -e`? – Charlie Parker Jan 20 '20 at 19:22
  • 2
    Why doing it the easy way if it can be done so much more cumbersome! – Peter Jun 24 '21 at 11:47
  • I tried to follow the instructions here but it said at "cookiecutter https://github.com/conda/cookiecutter-conda-python.git" that git is not installed. I tried conda install git but I got the same message. – Glxblt76 Sep 24 '21 at 16:15
  • I actually like this approach better than `conda develop` because you can also include your own modules in environment YAML files and more. Also, upgrading to a fully-fledged public module is straightforward this way. – Martin Hepp Nov 09 '21 at 10:03
  • Addendum: Using `pip` in conda environments works, but there are lots of reports of potential issues with this. It seems to work okay for a new environment, but updating may create inconsistencies. Personally I try to avoid pip inside a conda environment. – Martin Hepp Nov 09 '21 at 10:03
  • Addendum 2: There is a request to deprecate `conda develop` from `conda`, see https://github.com/conda/conda-build/issues/4251 – Martin Hepp Nov 12 '21 at 01:33
1

I had a wrapper function that I had to call multiple times on different scripts.So, I copied that file wrappers.py to the Anaconda site-packages folder. On my computer this was at: C:\ProgramData\Anaconda3\Lib\site-packages. Then, whenever I needed wrappers.py, I'd just import it in my scripts, like this:

import wrappers

If you want to make sure that the import was successful, you could either select Anaconda as your dev environment in your IDE and then invoke the Intellisense after import: from wrappers import (intellisense suggestions). Or you could also use IDLE:

>>> import wrappers
>>>(if you get this prompt, this means the import was successful).

Conda integration is only possible if you create custom channels. This is because conda searches for packages in the following locations(based on your OS) and to tell conda to search for your module at certain location(s), you must install conda-build to create a custom channel:

- https://repo.continuum.io/pkgs/main/win-64
- https://repo.continuum.io/pkgs/main/noarch
- https://repo.continuum.io/pkgs/free/win-64
- https://repo.continuum.io/pkgs/free/noarch
- https://repo.continuum.io/pkgs/r/win-64
- https://repo.continuum.io/pkgs/r/noarch
- https://repo.continuum.io/pkgs/pro/win-64
- https://repo.continuum.io/pkgs/pro/noarch
- https://repo.continuum.io/pkgs/msys2/win-64
- https://repo.continuum.io/pkgs/msys2/noarch

If you want to avoid creating a repo/channel, then a simple import as above should do the job. In addition, if you make any changes to your module and save it, you will always have the latest import in your scripts.

amanb
  • 5,276
  • 3
  • 19
  • 38