1

After trying to deal with relative imports and reading the many StackOverflow posts about it, I'm starting to realize this is way more complicated than it needs to be.

No one else is using this code but me. I'm not creating a tool or an api, so I see no need to create a "package" or a "module". I just want to organize my code into folders, as opposed to having a 50 scripts all in the same directory. It's just ridiculous.

Basically, just want to have three folders and be able to import scripts from wherever I want. One folder that contains some scripts with utility functions. Another folder has the majority of the code, and another folder that contains some experiments (I'm doing machine learning research). 

/project/code/
/project/utils/
/project/experiments/

All I need to do is import python files between folders.

One solution I have found is putting __init__.py files in each directory including the root directory where the folders live. But I then need to run my experiments in the parent of that directory.

/experiment1.py
/experiment2.py
/project/__init__.py
/project/code/__init__.py
/project/utils/__init__.py

So the above works. But then I have two problems. My experiments don't live in the same folder as the code. This is just annoying, but I guess I can live with it. But the bigger problem is that I can't put my experiments different folders:

/experiments/experiment1/something.py
/experiments/experiment2/something_else.py
/project/__init__.py
/project/code/__init__.py
/project/utils/__init__.py

I suppose I could symlink the project directory into each experiment folder, but thats ridiculous.

The other way of doing it is treating everything like a module:

/project/__init__.py
/project/code/__init__.py
/project/utils/__init__.py
/project/experiments/__init__.py
/project/experiments/experiment1/something.py
/project/experiments/experiment2/something_else.py

But then I have to run my experiments with python -m project.experiments.experiment1.something which just seems odd to me.

A solution I have found thus far is:

import imp
import os
currentDir = os.path.dirname(__file__)
filename = os.path.join(currentDir, '../../utils/helpful.py')
helpful = imp.load_source('helpful',filename)

This works, but it is tedious and ugly though. I tried creating a script to handle this for me, but then the os.path.dirname(__file__) is wrong.

Surely someone has been in my position trying to organize their python scripts in folders rather than having them all flat in a directory.

Is there a good, simple solution to this problem, or will I have to resort to one of the above?

dazedconfused
  • 1,312
  • 2
  • 19
  • 26
Chet
  • 18,421
  • 15
  • 69
  • 113
  • Just add to pythonpath and use import? – 101 Nov 11 '14 at 07:01
  • 2
    Regardless whether or not anyone else is going to be using your code that shouldn't stop you from doing it the right way. – kylieCatt Nov 11 '14 at 07:02
  • @IanAuld, agreed. So what is the right way? – Chet Nov 11 '14 at 19:22
  • One way is to use `setuptools` in development mode. https://pythonhosted.org/setuptools/setuptools.html#development-mode – kylieCatt Nov 11 '14 at 19:53
  • I'm sorry, but I got lost very quickly while reading that... I don't even have a `setup.py` file. I honestly know nothing about creating an distributing packages other than `pip install`. – Chet Nov 11 '14 at 19:57
  • [This link](https://pythonhosted.org/an_example_pypi_project/setuptools.html) should help. ALl you have to do once your `setup.py` file is ready is `python setup.py develop` – kylieCatt Nov 11 '14 at 20:02
  • you can try with the environment variable PYTHONPATH, put there all the folder that you want so their content can be imported from anywhere – Copperfield Mar 07 '16 at 21:57

2 Answers2

2

Running python files as a module seems also odd to me. You can put your experiments in a folder still by putting a main.py file on the root directory. So the folder tree would be the following;

/project/experiments
/project/code
/project/utils
/project/main.py

Call your experiments on your main.py file and make your imports on main.py file. The __init__.pys should also be inside each folder.

By this way, you won't need to run the py files as a python module. Also, the project will have a single entry point which is very useful in many cases.

aacanakin
  • 2,844
  • 4
  • 25
  • 42
  • this is a good idea. I can even do `from experiments.experiment1 import *` and then run `python -i main.py` and get all my variables to play with. @IanAuld, mentioned "the right way" so I'll wait to see what he says... – Chet Nov 11 '14 at 19:35
0

So I found this tutorial very useful.

The solution I have come up with is as follows:

project/
    .gitignore
    setup.py
    README.rst
    MANIFEST.in
    code/
        __init__.py
        something.py
        tests/
            __init__.py
            tests.py
        utils/
            __init__.py
            utils.py
    experiments/
        experiment1/
            data.json
            experiment1.py
        experiment2/
            data.json
            experiment2.py

Then I run python setup.py develop in order symlink my code so I can import it anywhere else (you can unlink with python setup.py develop --uninstall).

I still haven't decided whether I like my experiments to live inside the project folder or outside. I don't think it really matters since this code is only for my personal use. But I suppose it would be proper for it to live outside...

Chet
  • 18,421
  • 15
  • 69
  • 113
  • The link you provided has an expired redirect that directs [here](http://python-packaging.readthedocs.org/en/latest/minimal.html), according to Wayback Machine. – Keyacom Sep 17 '22 at 08:54