1

As far as I can tell, if a directory path contains dots then pkgutil.iter_modules breaks and appears to find modules where none exist.

i.e.

/var/www/www.example.org/src

is treated differently to

/var/www/wwwexampleorg/src

Does this make sense to anyone? I've been debugging a problem around this for 8 hours now and I'm kind of stuck about how to work out what is wrong. Does anyone have any insight into how pkgutil.iter_modules works and if it would break on directory paths which includes dots?

I will admit to being out of my depth on this one so not even sure how to write some sample code to demonstrate the problem. I decided the best way to explain it was briefly and to the point.

I'm using Python 3.4 on Ubuntu.

EDIT:

Apologies this is far from minimal. I'm sorry but the most effective way for me to show this is the sequence of actions that anyone can follow to reproduce:

ubuntu@app:~$ mkdir test
ubuntu@app:~$ cd test
ubuntu@app:~/test$ wget https://github.com/mattupstate/overholt/archive/master.zip
--2014-06-24 08:17:13--  https://github.com/mattupstate/overholt/archive/master.zip
Resolving github.com (github.com)... 192.30.252.128
Connecting to github.com (github.com)|192.30.252.128|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/mattupstate/overholt/zip/master [following]
--2014-06-24 08:17:13--  https://codeload.github.com/mattupstate/overholt/zip/master
Resolving codeload.github.com (codeload.github.com)... 192.30.252.144
Connecting to codeload.github.com (codeload.github.com)|192.30.252.144|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/zip]
Saving to: ‘master.zip’

    [   <=>                                                                                                                                    ] 218,165      386KB/s   in 0.6s   

2014-06-24 08:17:14 (386 KB/s) - ‘master.zip’ saved [218165]

ubuntu@app:~/test$ unzip master.zip 



(output not shown................unzips master.zip)


ubuntu@app:~/test$ ls
master.zip  overholt-master
ubuntu@app:~/test$ cd overholt-master/
ubuntu@app:~/test/overholt-master$ source ~/venv2.7/bin/activate
(venv2.7)ubuntu@app:~/test/overholt-master$ ls
alembic  alembic.ini  Berksfile  docs  LICENSE  manage.py  overholt  README.md  requirements.txt  tests  Vagrantfile  wsgi.py
(venv2.7)ubuntu@app:~/test/overholt-master$ pip install -r requirements.txt 

(................installs all requirements)



(venv2.7)ubuntu@app:~/test/overholt-master$ python wsgi.py 
 * Running on http://0.0.0.0:5000/
 * Restarting with reloader


(................YAY WORKS!, NOW TO RENAME ONE OF THE PARENT DIRECTORIES......)


^C(venv2.7)ubuntu@app:~/test/overholt-master$ cd ..
(venv2.7)ubuntu@app:~/test$ cd ..
(venv2.7)ubuntu@app:~$ mv test test.test
(venv2.7)ubuntu@app:~$ cd test.test/
(venv2.7)ubuntu@app:~/test.test$ cd overholt-master/
(venv2.7)ubuntu@app:~/test.test/overholt-master$ python wsgi.py 


(................FAILS......)

Traceback (most recent call last):
  File "wsgi.py", line 15, in <module>
    '/api': api.create_app()
  File "/home/ubuntu/test.test/overholt-master/overholt/api/__init__.py", line 23, in create_app
    register_security_blueprint=register_security_blueprint)
  File "/home/ubuntu/test.test/overholt-master/overholt/factory.py", line 44, in create_app
    register_blueprints(app, package_name, package_path)
  File "/home/ubuntu/test.test/overholt-master/overholt/helpers.py", line 26, in register_blueprints
    m = importlib.import_module('%s.%s' % (package_name, name))
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/home/ubuntu/test.test/overholt-master/overholt/api/stores.py", line 13, in <module>
    from ..tasks import send_manager_added_email, send_manager_removed_email
  File "/home/ubuntu/test.test/overholt-master/overholt/tasks.py", line 12, in <module>
    celery = create_celery_app()
  File "/home/ubuntu/test.test/overholt-master/overholt/factory.py", line 52, in create_celery_app
    app = app or create_app('overholt', os.path.dirname(__file__))
  File "/home/ubuntu/test.test/overholt-master/overholt/factory.py", line 44, in create_app
    register_blueprints(app, package_name, package_path)
  File "/home/ubuntu/test.test/overholt-master/overholt/helpers.py", line 26, in register_blueprints
    m = importlib.import_module('%s.%s' % (package_name, name))
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named overholt





(venv2.7)ubuntu@app:~/test.test/overholt-master$ 



(................NOW TO IDENTIFY THE CULRPIT??......)

(venv2.7)ubuntu@app:~/test.test$ cd overholt-master/
(venv2.7)ubuntu@app:~/test.test/overholt-master$ ls
alembic  alembic.ini  Berksfile  docs  LICENSE  manage.py  overholt  README.md  requirements.txt  tests  Vagrantfile  wsgi.py
(venv2.7)ubuntu@app:~/test.test/overholt-master$ cd overholt/
(venv2.7)ubuntu@app:~/test.test/overholt-master/overholt$ ls
api      core.pyc    factory.pyc  forms.pyc  helpers.py   __init__.py   manage         middleware.pyc  models.pyc  services.py   settings.py   stores    tasks.pyc
core.py  factory.py  forms.py     frontend   helpers.pyc  __init__.pyc  middleware.py  models.py       products    services.pyc  settings.pyc  tasks.py  users
(venv2.7)ubuntu@app:~/test.test/overholt-master/overholt$ cat helpers.py
# -*- coding: utf-8 -*-
"""
    overholt.helpers
    ~~~~~~~~~~~~~~~~

    overholt helpers module
"""

import pkgutil
import importlib

from flask import Blueprint
from flask.json import JSONEncoder as BaseJSONEncoder


def register_blueprints(app, package_name, package_path):
    """Register all Blueprint instances on the specified Flask application found
    in all modules for the specified package.

    :param app: the Flask application
    :param package_name: the package name
    :param package_path: the package path
    """
    rv = []
    for _, name, _ in pkgutil.iter_modules(package_path):
        m = importlib.import_module('%s.%s' % (package_name, name))
        for item in dir(m):
            item = getattr(m, item)
            if isinstance(item, Blueprint):
                app.register_blueprint(item)
            rv.append(item)
    return rv

(venv2.7)ubuntu@app:~$
Duke Dougal
  • 24,359
  • 31
  • 91
  • 123
  • 2
    Can you give a [minimal example](http://stackoverflow.com/help/mcve) of how it breaks, so we can set up the same situation on our own machines and replicate the issue? – user2357112 Jun 24 '14 at 08:13
  • The Python package system uses dots as path separators. It's therefore a bad idea to have dots in your file or directory names when they are used to build a package structure. I don't know about your specific problem (http://stackoverflow.com/questions/1828127/how-to-reference-python-package-when-filename-contains-a-period deals with filenames, not directory names), but I strongly suspect that it's similar to this. – Tim Pietzcker Jun 24 '14 at 08:53
  • That is, as you've said, quite far from minimal. I'd really rather not install all this stuff and spawn a web server just to replicate the issue. Can't you write up something on the scale of a 7-line, 1-file example without all these complicating factors? – user2357112 Jun 24 '14 at 08:55
  • @user2357112 thanks for your interest, I'm not sure I would be able to write a short example because I've tracked the bug down in someone else code and I can't say I have the chops to write and example around it. I suspect TimPietzcker's answer will suffice which is to say "don't have dots in directory names", which is kind of tedious but now that I know I can work around it. thanks again. – Duke Dougal Jun 24 '14 at 08:59

0 Answers0