20

I have a file structure like this:

project_folder/
     notebooks/
          notebook01.ipynb
          notebook02.ipynb
          ...
          notebookXY.ipynb
     module01.py
     module02.py
     module03.py

In .ipynb files inside notebook/ folder I want to import classes and functions from module01.py, module02.py and module03.py.

I have found answer in this question that it is possible using following lines of code inside every notebook and run those lines as first cell every time:

import os
import sys
module_path = os.path.abspath(os.path.join('..'))
if module_path not in sys.path:
    sys.path.append(module_path)

Is there please a better way for this? What if I have A LOT of .ipynb files inside notebooks/ folder, do I have to paste those lines of code at the beginning of every single one? Is there a better, more minimalist or cleaner way?

Community
  • 1
  • 1
PeterB
  • 2,234
  • 6
  • 24
  • 43

2 Answers2

6

Another solution is to move all your Python modules (.py files) into a folder and make them an installable package. If you pip install it into your current environment, you can then import the package into any notebook in that environment, regardless of folder structure.

So in your situation you could have:

project_folder/
  notebooks/
    notebook01.ipynb
    notebook02.ipynb
    ...
    notebookXY.ipynb
  my_package/
    __init__.py
    module01.py
    module02.py
    module03.py
  setup.py
  • __init__.py can just be an empty file, and tells Python "everything in this folder is part of a package"
  • For an explanation of what goes in setup.py see here.

A basic setup.py can be as simple as this:

import setuptools

setuptools.setup(
    name="my_package",
    version="0.0.1",
    description="A small example package",
    packages=setuptools.find_packages(),
    python_requires='>=3.7',
)

Install it:

cd project_folder
pip install [-e] .

Including the optional -e flag will install my_package in "editable" mode, meaning that instead of copying the files into your virtual environment, a symlink will be created to the files where they are.

Now in any notebook you can do:

import my_package

Or

from my_package.module01 import <some object>

David Norrish
  • 347
  • 2
  • 7
  • You should put the setup.py file as well in the my_package folder. In my case, it couldn't find the necessary files within the folder. If it is in the same folder it works perfectelly. Thanks a lot. Very helpful. – Manthano Jan 04 '21 at 19:41
  • Hi this is a really useful answer however when I try and install my_package in the same set up as above from one of the notebooks (notebook01.ipynb) it won't work since the working directory is in notebooks and not the parent project_folder. Is there a way to deal with this nicely? I.e. so all notebooks are created with the working directory set to project_folder or other solution? – A_Murphy Nov 04 '21 at 10:02
5

Try adding the project_folder to your PYTHONPATH environment variable. This will allow you to tell python to search that directory for imports.

You would do this in your user profile settings, or in your startup script - not in python. It's something that has to be set before python ever gets run.

aghast
  • 14,785
  • 3
  • 24
  • 56
  • how can I do this please? – PeterB May 08 '17 at 14:19
  • How are you running the `ipynb` files? From a command line, by clicking with a mouse, something else? – aghast May 08 '17 at 16:39
  • 6
    clicking in Jupyter Notebook, I can do it by running cell with code: `sys.path.append(os.path.dirname(os.getcwd()))` but it does not solve my problem permanently, just until I restart kernel, it's almost the same as original solution in my Q body – PeterB May 08 '17 at 17:12
  • What OS are you using? – aghast May 08 '17 at 17:22
  • I am using Ubuntu – PeterB May 08 '17 at 17:22
  • Try doing `export PYTHONPATH="$HOME/path/to/project_folder:$PYTHONPATH"` in your ~/.profile file. (Google returns a lot of hits for this.) – aghast May 08 '17 at 17:25