1

I have a typical project structure that looks as follows:

EngineEmulator
    src
        ship
            engine
                emulator
                mapping
            tests
                emulator
                mapping
        utils
            common

    doc
       ....
    tools
       ....
    setup.py
    MANIFEST.in
    setup.cfg
    README.rst

My setup.py looks as follows:

from setuptools import setup, find_packages
setup(
   name='Engine',
   version=1.0.0,
   description='Engine Project',       
   package_dir={'': 'src'},
   packages=find_packages(
    'src',
    exclude=["*.tests", "*.tests.*", "tests.*", "tests"]),
   install_requires =['pycrypto', 
                      'kombu >=1.1.3'],
   author='Demo',
   author_email='demo@eliza.net'
   license='MIT',
   classifiers=[
    'Topic :: Demo Engine',
    'Development Status:: 3 - Iteration',
    'Programming Language :: Python -2.6'
]

)

My setup.cfg looks as follows:

[egg_info]
tag_build = .dev
tag_svn_revision = 1

[rotate]
#keep last 15 eggs, clean up order
match = .egg
keep = 15

And My MANIFEST.in looks as follows:

include README.rst
recursive-include src/ship/Engine
prune src/utils
prune src/ship/tests
prune tools/

When I run python setup.py bdist_egg and python setup.py bdist_rpm I get the egg file and two rpm files generated (noarch.rpm and src.rpm).

In my destination machine when I run easy_install <generated egg file> my eg.info file gets copied over but the source files don't get copied over to /usr/lib/python2.6/site-packages. I was expecting I would have a directory called Engine.

Can anybody point out what I am doing wrong? Thanks in advance.

as3rdaccount
  • 3,711
  • 12
  • 42
  • 62
  • @ArunaDev Assuming, your project is pure Python one (no C or C++ code in your project), I would recommend using `sdist` and not `egg`. I also assume, you install in the same OS as you create the egg. Learn using `pip` and stop using `easy_install` whenever possible. `pip` does much better service. You shall be able `$ pip install ` to install from egg, sdist or other formats. – Jan Vlcinsky Jul 25 '14 at 07:54
  • pip does not support egg. sdist would work but we want some C code to be prebuilt before handing over to customers. Also for other reasons that I won't go into we want to ship only .pyc files. – as3rdaccount Jul 25 '14 at 19:15
  • @ArunaDev Sorry, I was wrong with `pip` and eggs. You might find using `wheel` format handy - it is the faster way to install a package. But it seems to include source python files, so it might not meet your criteria. Anyway, having "only pyc" is not so big win, once I mistakenly deleted my py files and managed get usable source from pyc files within an hour of googling, testing, decompiling and editing. – Jan Vlcinsky Jul 25 '14 at 20:25
  • Yes it is not so much of a reason for easiness in decompiling...it is much more ease for support stuff. – as3rdaccount Jul 25 '14 at 20:37

1 Answers1

1

Try to keep things as simple as possible.

Quick check with sdist

Try this:

$ python setup.py sdist

It shall create source distribution file for your package.

It is in zip format, so unpack it and check, if there are all expected files inside present.

If not, you have to find the reason, why expected files are missing in your distribution.

Checking things step by step (and simplifying)

Do you use .py extension?

May be stupid question, but in your file listing I do not see any py files inside of src tree.

In case you have there just files without .py extension, find_packages will not find anything.

Where do you have your __init__.py files located?

Let us know, where the files are:

$ cd src
$ find . -name "*.py"

If you miss __init__.py, find_packages will not find whole package.

Remove utils package

Why do you have it there?

Better have it installed out of your source code you develop or move it subdirectory in your project root.

This will render prune src/utils unnecessary in your MANIFEST.in.

Put into MANIFEST.in only what must be there

If you read doc for MANIFEST.in, it states, what files are included automatically (all, what mentioned in arguments of setup function, so in your case all python source files returned by find_packages).

For this reason, you shall remove recursive-include src/shop/Engine as it shall be already included by setup call.

Remove prune lines.

  • src/utils shall not be in your source tree - it is just messing things up.
  • tools is not to be included, so there is no need to prune it.
  • src/ship/tests can be there, it will not harm, if you keep these files in the destribution.

Assert, what packages were found

Make sure, your setup get proper names for packages.

For this purpuse, you can call find_package sooner and assert it containts, what you expect.

(temporarily) remove setup.cfg

Just to keep things simpler.

Proposed project reorganization

You shall have file structure in similar manner as follows:

src/ship/__init__.py
src/ship/engine/__init__.py
src/ship/engine/emulator/__init__.py
src/ship/engine/emulator/module.py
src/ship/engine/emulator/module2.py
src/ship/engine/mapping/other.py
src/ship/engine/mapping/another.py
src/ship/tests/__init__.py
src/ship/tests/emulator/__init__.py
src/ship/tests/emulator/test_module.py
src/ship/tests/emulator/test_module2.py
src/ship/tests/mapping/__init__.py
src/ship/tests/mapping/test_other.py
src/ship/tests/mapping/test_another.py
doc
doc/index.rst
tools
tools/knife.py
setup.py
MANIFEST.in
README.rst

setup.py

from setuptools import setup, find_packages

packages=find_packages("src")
assert "ship.engine" in packages
assert "ship.engine.emulator" in packages
assert "ship.engine.mapping" in packages
#etc

install_requires =['pycrypto', 'kombu>=1.1.3'] #watch the spaces around `>=`, shall not be there

setup(
    name="Engine",
    package_dir={'': 'src'},
    packages=packages,
    install_requires=install_requires
)

MANIFEST.in

include README.rst

Conclusions

It might happen, that running

$ python setup.py sdist

would fail on asserts. This is sign, some of expected files are missing. Check that.

After you make your project living in simple way, you might add more details around (and do it step by step to be sure, you do not break something).

Jan Vlcinsky
  • 42,725
  • 12
  • 101
  • 98
  • Jan Thanks a lot for your answer! Yes my src dir was having a__init__.py that I omitted in the code in question. And sdist contents does contain the modules. assert passes too. I made all the changes you suggested. Then I run easy_install `python setup.py bdist_egg` but when on the destination machine I run easy_install none of the source files are copied over to /usr/lib/python2.6/site-packages/. I was expecting a directory 'Engine', am I correct (since that is the project name I specified)? There is just a Engine-1.0.0-py2.7.egg directory. – as3rdaccount Jul 24 '14 at 22:48
  • @ArunavDev Try `$ pip freeze` and check, if the package is listed. Try `pip show Engine`, this shall show you, where is the package installed. The most important is, if you are able importing the module. Creation of some directories is secondary topic. Btw, do you use virtualenv? If so (and it is highly recommended), the packages get installed into virtulanev directories and not to the system. – Jan Vlcinsky Jul 25 '14 at 07:47
  • @ArunavDev Btw, there shall be no `__init__.py` package in `src` directory. `__init__.py` must be only in directories, which are packages and `src` is not supposed to be a package, but parent directory of package directories. – Jan Vlcinsky Jul 25 '14 at 07:50
  • Thanks Jan. pip freeze shows the Engine==1.0.0 and pip show Engine shows the egg file location in site package. Looks like egg file inside contains the Engine and EGG-INFO file. This is defeating my ultimate goal since Engine is not in the default PYTHONPATH anymore. – as3rdaccount Jul 25 '14 at 19:17