How to import a package module from inside a Flask route?

Question

I am migrating an app I have written to the standard Python package format. As I have my app's modules divided over several directories, I used to import these by inserting their respective paths to the sys.path, but this is no longer an option as these modules also import their own respective dependencies from inside the package.

I am trying to load a set of modules from inside a route (@app.route) in Flask in a separate file.

@app.route("/interface")
def interface():
    moduleName = "Olympus.modules.%s.%s" % (category, module)
    print "Importing:", moduleName
    importedModule = __import__(moduleName)

(I have removed extranuous code in this function.) This function however, only yields ImportErrors, telling me "there is no module named ". There are __init__.py files in every (sub)directory and these modules import effortlessly elsewhere. Importing the modules from the containing directory's __init__.py results in the modules and their siblings (eg. all the scripts in the same category) being accessible from the script, but this requires all me tot hardcode all the modules in the init file.

The routes are located in a separate file as suggested by Flask (http://flask.pocoo.org/docs/patterns/packages/) which is imported by the init file of the app. When __name__ is printed it yields the proper, packaged name. The modules can also be imported individually at the top of the file, with no ill effects. Printing pkgutils.itermodules as suggested here inside the function yields an empty list.

What am I doing wrong SO? How do I import these modules into this Flask route?

EDIT: The code in question can be found here: https://github.com/HAN-Olympus/Olympus/tree/master/webapp EDIT: Here is the directory structure, I have pruned it somewhat, but it is still rather extensive.

.
├── Olympus
│   ├──__init__.py
│   ├── core
│   │   ├── Core.py
│   │   ├── __init__.py
│   │   ├── ModuleLoader.py
│   │   ├── Worker.py
│   ├── createMongoDatabase.js
│   ├── default.conf
│   ├── __init__.py
│   ├── lib
│   │   ├── Article.py
│   │   ├── Chemical.py
│   │   ├── Collection.py
│   │   ├── Config.py
│   │   ├── Controls.py
│   │   ├── Gene.py
│   │   ├── __init__.py
│   │   ├── ...
│   │   ├── TOXNETResult.py
│   ├── modules
│   │   ├── acquisition
│   │   │   ├── AcquisitionModule.py
│   │   │   ├── GeneOntology.py
│   │   │   ├── __init__.py
│   │   │   ├── PubMed.py
│   │   │   ├── TOXNET.py
│   │   │   ├── UniProt.py
│   │   │   ├── WormBase.py
│   │   ├── __init__.py
│   │   ├── interface
│   │   │   ├── __init__.py
│   │   │   ├── InterfaceModule.py
│   │   │   ├── LaTeX.py
│   │   │   ├── PlainHTML.py
│   │   │   ├── StyledHTML.py
│   │   ├── interpretation
│   │   │   ├── __init__.py
│   │   │   ├── InterpretationModule.py
│   │   │   ├── Sort.py
│   │   │   ├── Tail.py
│   │   └── visualization
│   │       ├── FancyTable.py
│   │       ├── __init__.py
│   │       ├── Table.py
│   │       ├── VisualizationModule.py
│   ├── olympus.conf
│   ├── README.md
│   ├── requirements.txt
│   └── webapp
│       ├── __init__.py
│       ├── routes.py
│       ├── start.py
│       ├── svglib.py
│       └── templates
│           ├── ...

Are you certain you can import the modules *directly* without using `__import__`? There is nothing special about Flask routes when it comes to importing. — Martijn Pieters, May 26 '14 at 09:55
Putting the imports at the top of the routes.py file (wherein the routes are located) Achieves the desired effect, however, I would like the modules to be loaded dynamically, which does not happen when you import the routes file. — Stephan Heijl, May 26 '14 at 10:50
I have tried this with `import Project.modules.category.module`. This yields `ImportError: No module named modules.category.module` . I should note that I have migitated the problem by loading the modules dynamically in the init file, but it still seems strange to me that I cannot import package modules inside the routes. — Stephan Heijl, May 26 '14 at 10:58
Sounds like you have a `Projects.py` or `Projects` package somewhere *else* that is masking the import. What does `import Projects; print Projects.__file__` tell you is imported? — Martijn Pieters, May 26 '14 at 11:03
@StephanHeijl You shall tell us a bit more. Could you show tree of directories and files you use? Martijn might be right, that there is some Projects.py or directory making problems. — Jan Vlcinsky, May 26 '14 at 11:07
Edited the question to include a directory structure. I should note that I used the names "Project" etc to make the question more generic. This would refer to `Olympus.modules.acquisition.PubMed`, for example. — Stephan Heijl, May 26 '14 at 11:24
@MartijnPieters This yields `Olympus/__init__.pyc`. I have attempted the same with `import Olympus.modules ; print Olympus.modules.__file__`, but it throws an ImportError (no module named modules) . Removing the empty `modules` directory from the webapp directory yielded no results. — Stephan Heijl, May 26 '14 at 11:31
@StephanHeijl: that's a *relative path*; relative to the current working directory. Are you certain that that's the right package to import? Clearly there is no `modules` package or module inside of that. — Martijn Pieters, May 26 '14 at 11:33
I have included a link to the code in the post. The modules package is located in the Olympus package (topmost directory). Importing this way works fine in all other scripts, the only place I encounter this issue is when I try to import in a route. — Stephan Heijl, May 26 '14 at 15:24

Jan Vlcinsky · Answer 1 · 2014-05-26T08:59:42.123

Maintain all modules as standard modules

Keep it simple and install your modules to import into your Flask environment.

Make all the modules complete (incl. setup.py)

To make your modules installable, they must include setup.py installation script. Check documentation, how to do this.

Installing modules into Flask app

From your Flask app directory, you can install other modules by:

$ pip install ../other_module_dir

and do so repeatedly.

Installing modules in "edit" mode

If you want to have these installed in the way, that you use the latest version, you can use "edit" installation mode

$ pip install -e ../other_module_dir

With this approach, if you modify code in your ../other_module_dir project, next time it will get imported in updated version.

Automate installation of modules using `requirements.txt`

To automate installations, you can finally dump all the dependencies into file (usually) named requirements.txt:

$ pip freeze > requiremetns.txt

edit it (remove, what does not belong there, and finally you may install all dependent modules in one step:

$ pip install -r requirements.txt

Conclusion

The most demanding (but not so difficult) is to make your modules complete with setup.py. Once you manage this (and this skill will pay back many times later on), your whole scenario of import will become trivial, just following names of your modules you import and have them available (I recommend using absolute imports).

Note: Use virtualenv for this kind of work, it is not required, but it will make your work much simpler.

I should have made more clear that these modules exist in a package. This seems needlessly complicated for something that should be achieved by Python's built in import management with a simple `__import__("Project.modules.category.module", fromlist=["Project.modules.category"])` — Stephan Heijl, May 26 '14 at 10:53