0

I am trying to work on improvements to a pip installed package developed by our team. I am unable to import my improved and additional modules for thoroughly testing before integration into new version. I understand this is likely namespace issues, but i'm unable to resolve them. I've read what I can find, including this excellent blog on import traps.

I am doing my testing and applications for the library methods in Jupyter. In a failed attempt to avoid collision, I have this renamed the top package dir:

new_package/
  setup.py
  __init__.py
  new_package_dir/
    __init__.py
    cool_things.py
    dir1/
      __init__.py
      foo.py
    dir2/
      __init__.py
      bar.py

i have no main() methods. we are using python 3.6. the package was setup so that the top level __init__.py file contains: from dir1 import * and from dir2 import *. the other __init__.py files are empty.

in jupyter, i have tried to 

import sys
sys.path.append('absolute_path_to_new_package_dir/')
import new_package_dir.dir1.foo as foo

as well as several other variants. I get the error ModuleNotFoundError: No module named 'new_package'.

appreciate any advice. my problems seems nearly identical to one described in the blog above, except i don't want to execute main(), i want the library available for working in Jupyter.

Renée
  • 455
  • 2
  • 7
  • 15

1 Answers1

0

I'm now able to answer my own question, by piecing together various other stackoverflow answers and blog posts. In the end, the solution was to:

  • create a virtual environment,
  • create a development version of the working package within that environment, and
  • create a jupyter/ipython kernel for the the environment to use in JupyterHub.

Specifically, this is how I achieved those things and the hiccups along the way:

Creating the Virtual Environment

This is all standard conda virtual environments. Install virtualenv. If conda is already installed, use

conda install virtualenv

pip install will not work in this case. Then create a directory 'virtualenvs' in your home directory to store the environment. The directory name and environment name don't matter.

virtualenv ~/virtualenvs/my_env

creates a virtual environment, my_env. Now, activate that with 

source ~/virtualenvs/my_env/bin/activate

The prompt will show the new environment.

Create Development Package

Navigate to the package you want to develop in and run 

python setup.py develop

In my case, this chugged for quite a long time and gave endless warnings about numpy. I didn't investigate this further because in the end it worked, and it seems others reported similar issues with develop and numpy.

When its done, you should get a message like 

Using <your_home>/virtualsenvs/my_env/lib/python3.6/site-packages
Finished Processing dependencies for way-cool-package==0.01

You can navigate to that location and confirm existence of an egg.lnk file, and you can also check the package is 'installed' with 

pip freeze

At this point, with the python interpreter you can test that your module is accessible. I added a new print_hello method to my module and then checked that I could import that. The first time I did all this, I didn't have the order correct, and as a result, python was only importing the site-installed package, not my new enhancements.

We work in JupyterHub a lot, so this wasn't good enough for our team. If you try to use ipython on the command line here, it will nag. Next step..

Create Jupyter Kernel

Within the new virtual environment, check if ipykernel module is installed by using pip freeze. Otherwise, 

pip install ipykernel

Create a new Jupyter kernel with the name new_kernel by

python -m ipykernel install --user --name=new_kernel

This will install the kernel in your home directory and report

Installed kernelspeck new_kernel in <your_home>/.local/share/jupyter/kernels/new_kernel

Now, refresh your JupyterHub Notebook, open an new one, and you should have 'new_kernel' as an option.

The Last Gotcha

The last gotcha I encountered was in a colleague's package on which I have a dependency. They used __file__ in their code to create their doc string. When trying to use my own modules in the new kernel, I hit a 'NotADirectoryError' which puzzled me. After all, their code was pip-installed site-wide and had no issues. Then, I noticed in the setup tools documentation that instead of using references like __file__, which make an assumption about the path, you should use pkg_resource.resouce_string(). I made that change to their code, and all was perfect.

Here are some of the resources that I used:

Community
  • 1
  • 1
Renée
  • 455
  • 2
  • 7
  • 15