2

UPDATE: When installing both packages using setup.py alone they install just fine. When extracting the tarballs generated by sdist and installing them the same error occurs. This means that the problem is somewhere inside setuptools I guess.

I developed two projects that have two namespace packages: testsuite and testsuite.prettyprint. Both of these namespace packages' __init__.py contain:

__import__('pkg_resources').declare_namespace(__name__)

Here's the setup.py for testsuite.prettyprint.outcomes:

import pkgutil
from setuptools import setup


def get_packages():
    return [name for _, name, is_package in pkgutil.walk_packages('.') if name.startswith('testsuite') and is_package]

dependencies = ['nose2>=0.4.6', 'colorama>=0.2.5']

setup(
    name='testsuite-prettyprint-outcomes',
    version='0.1.0-alpha.1',
    packages=get_packages(),
    url='',
    license='BSD3',
    author='Omer Katz',
    author_email='omer.drow@gmail.com',
    description='testsuite-prettyprint-outcomes is a nose2 plugin that prettyprints test outcomes.',
    namespace_packages=['testsuite', 'testsuite.prettyprint'],
    install_requires=dependencies
)

and here is the setup.py for testsuite.prettyprint.traceback:

import pkgutil
import sys

from setuptools import setup


def get_packages():
    return [name for _, name, is_package in pkgutil.walk_packages('.') if name.startswith('testsuite') and is_package]

dependencies = ['nose2>=0.4.6', 'pygments>=1.6']

if sys.platform == 'win32':
    dependencies.append('colorama>=0.2.5')

setup(
    name='testsuite-prettyprint-traceback',
    version='0.1.0-alpha.2',
    packages=get_packages(),
    url='',
    license='BSD3',
    author='Omer Katz',
    author_email='omer.drow@gmail.com',
    description='testsuite-prettyprint-traceback is a nose2 plugin that prettyprints traceback on failures and errors.',
    namespace_packages=['testsuite', 'testsuite.prettyprint'],
    install_requires=dependencies
)

When installing them both it refuses to install one:

pip install testsuite-prettyprint-outcomes testsuite-prettyprint-traceback --use-mirrors
Downloading/unpacking testsuite-prettyprint-outcomes
  Downloading testsuite-prettyprint-outcomes-0.1.0-alpha.1.tar.gz
  Running setup.py egg_info for package testsuite-prettyprint-outcomes

Downloading/unpacking testsuite-prettyprint-traceback
  Downloading testsuite-prettyprint-traceback-0.1.0-alpha.2.tar.gz
  Running setup.py egg_info for package testsuite-prettyprint-traceback

Downloading/unpacking nose2>=0.4.6 (from testsuite-prettyprint-outcomes)
  Running setup.py egg_info for package nose2

    warning: no previously-included files matching '__pycache__' found anywhere in distribution
    warning: no previously-included files matching '*~' found anywhere in distribution
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
Downloading/unpacking colorama>=0.2.5 (from testsuite-prettyprint-outcomes)
  Downloading colorama-0.2.5.zip
  Running setup.py egg_info for package colorama

Downloading/unpacking pygments>=1.6 (from testsuite-prettyprint-traceback)
  Downloading Pygments-1.6.tar.gz (1.4MB): 1.4MB downloaded
  Running setup.py egg_info for package pygments

Downloading/unpacking six>=1.1,<1.2 (from nose2>=0.4.6->testsuite-prettyprint-outcomes)
  Running setup.py egg_info for package six

Installing collected packages: testsuite-prettyprint-outcomes, testsuite-prettyprint-traceback, nose2, colorama, pygments, six
  Running setup.py install for testsuite-prettyprint-outcomes
    Skipping installation of /home/omer/.virtualenvs/test/lib/python3.2/site-packages/testsuite/__init__.py (namespace package)
    Skipping installation of /home/omer/.virtualenvs/test/lib/python3.2/site-packages/testsuite/prettyprint/__init__.py (namespace package)

    Installing /home/omer/.virtualenvs/test/lib/python3.2/site-packages/testsuite_prettyprint_outcomes-0.1.0_alpha.1-py3.2-nspkg.pth
  Running setup.py install for testsuite-prettyprint-traceback
    error: package directory 'testsuite/prettyprint/outcomes' does not exist
    Complete output from command /home/omer/.virtualenvs/test/bin/python3.2 -c "import setuptools;__file__='/home/omer/.virtualenvs/test/build/testsuite-prettyprint-traceback/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-12l9lq-record/install-record.txt --single-version-externally-managed --install-headers /home/omer/.virtualenvs/test/include/site/python3.2:
    running install

running build

running build_py

creating build

creating build/lib

creating build/lib/testsuite

copying testsuite/__init__.py -> build/lib/testsuite

creating build/lib/testsuite/prettyprint

copying testsuite/prettyprint/__init__.py -> build/lib/testsuite/prettyprint

error: package directory 'testsuite/prettyprint/outcomes' does not exist

----------------------------------------
Command /home/omer/.virtualenvs/test/bin/python3.2 -c "import setuptools;__file__='/home/omer/.virtualenvs/test/build/testsuite-prettyprint-traceback/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-12l9lq-record/install-record.txt --single-version-externally-managed --install-headers /home/omer/.virtualenvs/test/include/site/python3.2 failed with error code 1 in /home/omer/.virtualenvs/test/build/testsuite-prettyprint-traceback
Storing complete log in /home/omer/.pip/pip.log

I can't figure out what is wrong. Even if you change the order of installation it says it can't find the other.

shuttle87
  • 15,466
  • 11
  • 77
  • 106
the_drow
  • 18,571
  • 25
  • 126
  • 193
  • @abarnert Installing any of them alone works. Installing both does **not**. – the_drow Jun 27 '13 at 09:13
  • What did you enter into the `namespace_packages` entry for the `setup.py` files? – Martijn Pieters Jun 27 '13 at 09:28
  • @MartijnPieters both contain ['testsuite', 'testsuite.prettyprint']. I checked and when I sdist them namespace_packages.txt contains both. – the_drow Jun 27 '13 at 09:33
  • Any chance you can bump your requirements up to Python 3.3 and just rely on [PEP 420](http://www.python.org/dev/peps/pep-0420/) instead of building explicit namespace packages? I'm guessing no, but… hacking things up to remove the `__init__.py` files and the `namespace_packages` in `setup.py` and so on, it worked with 3.3. (But of course with 3.2, it installed and then couldn't be used, which is not exactly friendly…) – abarnert Jun 27 '13 at 10:03
  • @abarnert Not really no. – the_drow Jun 27 '13 at 11:17

3 Answers3

3

After installing one of your packages and downloading the other…

You're not including testsuite/__init__.py and testsuite/prettyprint/__init__.py in the source files.

The docs on Namespace Packages the setuptools/pkg_resources way explains:

Note, by the way, that your project's source tree must include the namespace packages' __init__.py files (and the __init__.py of any parent packages), in a normal Python package layout.

If you don't actually install these files, they don't do any good. You just end up with a testsuite with nothing in it but prettyprint, and that has nothing in it but outcomes, so testsuite and testsuite.prettyprint are not packages at all, much less namespace packages.

Dima Tisnek
  • 11,241
  • 4
  • 68
  • 120
abarnert
  • 354,177
  • 51
  • 601
  • 671
  • How do I include them both? – the_drow Jun 27 '13 at 09:34
  • 1
    Hold on, let me figure out why they're not being included, and I'll edit the answer. – abarnert Jun 27 '13 at 09:38
  • I just checked and get_packaged returns all packages correctly. – the_drow Jun 27 '13 at 09:43
  • 1
    The setup.py files look right for this type of setup… but I've only ever used this type with egg installs (`easy_install-3.3 .` or `pip-3.3 install --egg .`). And when I do that, I _do_ get the `__init__.py` files; it's just that non-egg installs skip them—with an explicit `Skipping installation of …/testsuite/__init__.py (namespace package)` message. – abarnert Jun 27 '13 at 09:48
  • Why do I have to install using egg? I upload using setup.py build sdist upload – the_drow Jun 27 '13 at 09:50
  • As I said, I haven't used pkg_resources-style namespace packages without eggs, and for some reason it makes a difference, and I don't know why. I mean, I know what's different—with eggs, each one has its own `testsuite` directory that both get added to the path; without, they install to the same place. But I don't know why `pip` doesn't install the `__init__.py` iff not present, which seems like the obvious thing to do, and which would solve the problem. I'll take a deeper look tomorrow, but I may not be able to help. Hopefully my answer will at least trigger someone who knows better… – abarnert Jun 27 '13 at 09:53
  • A quick search of pip namespace turned up a few other questions about problems with namespace packages, pip, and non-egg installs, like [this one](http://stackoverflow.com/questions/13400291/namespace-packages-and-pip-install-e). None of them seems the same as this problem, but… a bit more searching is probably worth trying. – abarnert Jun 27 '13 at 09:55
  • It still doesn't install even if I use eggs. – the_drow Jun 27 '13 at 09:55
  • 1
    Aha. Check out the comments [here](https://github.com/pypa/pip/issues/3#issuecomment-1659959) and following. It looks like `pip` intentionally doesn't install `__init__.py` for namespace packages for non-egg installs, and instead uses a different mechanism. I don't know what the _answer_ is, but at least that tells us where to look. (Unfortunately, I have to go now, but I'll see what I can turn up tomorrow if nobody else helps first.) – abarnert Jun 27 '13 at 09:59
  • Even if I do install it using --egg it still complains the same error. It also cannot be installed using easy_install. Same error. – the_drow Jun 27 '13 at 11:29
  • Maybe I should distribute them with bdist_egg and not sdist. I'm not sure if that would help. – the_drow Jun 27 '13 at 12:11
  • @the_drow: As I said in the comments above, the problem seems to be with `pip`, so I'm not surprised that installing with `setup.py` is a problem. But I don't know if `pip` has solved the problem yet, or if the devs have found a workaround. – abarnert Jul 01 '13 at 23:12
  • The problem is not with pip because when I sdist, extract and use the extracted setup.py it produces the error as well. When I use setup.py from the project source it works just fine. – the_drow Jul 02 '13 at 19:32
0

The names of your packages look wrong. I just separated a project out into subpackages, and one thing I did differently was to make each name match the components of the namespace_packages.

So, for testsuite.prettyprint.outcomes:

setup(
       name='testsuite.prettyprint.outcomes',
       [...] ,
       namespace_packages=['testsuite', 'testsuite.prettyprint']
)

And for testsuite.prettyprint.traceback:

setup(
       name='testsuite.prettyprint.traceback',
       [...] ,
       namespace_packages=['testsuite', 'testsuite.prettyprint']
)

For this to work properly, you'll need to provide __init__.py scripts, like you've already shown, for all the parent namespace_package levels (i.e. down to testsuite.prettyprint).

Good examples of namespace_packages in production packages can be found in the zope sub-packages.

e.g. See the zope.app.cache setup.py script, at http://svn.zope.org/zope.app.cache/trunk/setup.py?view=markup

Alex Leach
  • 1,199
  • 2
  • 12
  • 26
  • But then they won't be installable as testsuite-prettyprint-traceback. There is a reason I am specifying the package name. Is this the bug we're trying to track in setuptools? – the_drow Jul 08 '13 at 12:24
-2

I see you are using virtualenv. Usually when I encounter that error your are facing is because the packages don't work well with virtualenv.

Have you tried installing the packages on your main python install folder? (not in virtualenv)

I think this happens because some setup.py files make assumptions about the host environment and don't follow the setup.py best practices.

If you are still stuck give it a try.

Gabriel
  • 1,078
  • 1
  • 8
  • 15