49

I can't properly install the project package_fiddler to my virtual environment.

I have figured out that MANIFEST.in is responsible for putting the non-.py files in Package_fiddler-0.0.0.tar.gz that is generated when executing python setup.py sdist.

Then I did:

(virt_envir)$ pip install dist/Package_fiddler-0.0.0.tar.gz

But this did not install the data files nor the package to /home/username/.virtualenvs/virt_envir/local/lib/python2.7/site-packages.

I have tried many configurations of the setup arguments package_data, include_package_data and data_files but I seem to have used the wrong configuration each time.

Which configuration of package_data and/or include_package_data and/or data_files will properly install package_fiddler to my virtual environment?

Project tree

.
├── MANIFEST.in
├── package_fiddler
│   ├── data
│   │   ├── example.html
│   │   └── stylesheets
│   │       └── example.css
│   └── __init__.py
├── README.rst
└── setup.py

setup.py

from setuptools import setup


setup(
    name='Package_fiddler',
    entry_points={
    'console_scripts': ['package_fiddler = package_fiddler:main', ],},
    long_description=open('README.rst').read(),
    packages=['package_fiddler',])

MANIFEST.in

include README.rst
recursive-include package_fiddler/data *

Which configurations of setup.py(with code base above) have I tried?

Configuration1

Adding:

package_data={"": ['package_fiddler/data/*',]}

Configuration2

Adding:

package_data={"": ['*.html', '*.css', '*.rst']}

Configuration3

Adding:

include_package_data=True

Configuration4

Adding:

package_data={"": ['package_fiddler/data',]}

Removing:

packages=['package_fiddler',]

Configuration5 (Chris's suggestion)

Adding:

package_data={"data": ['package_fiddler/data',]}

Removing:

packages=['package_fiddler',]

Configuration 6

Adding: 

package_data={"": ['package_fiddler/data/*',]}

Removing:

packages=['package_fiddler',]

These configurations all result in no files at all being installed on /home/username/.virtualenvs/virt_envir/local/lib/python2.7/site-packages.

EDIT

Note to Toshio Kuratomi: In my original post I used the simplest tree structure where this problem occurs for clarity but in reality my tree looks more like the tree below. For that tree, strangely if I only put an __init__.py in stylesheets somehow all the data files in the texts folder are also installed correctly!!! This baffles me.

Tree 2 (This installs all data files properly somehow!!)

.
├── MANIFEST.in
├── package_fiddler
│   │── stylesheets
|   |     ├── __init__.py
|   |     ├── example.css  
|   |     └── other
|   |          └── example2.css
|   |__ texts
|   |     ├── example.txt  
|   |     └── other
|   |          └── example2.txt
│   └── __init__.py
├── README.rst
└── setup.py
Bentley4
  • 10,678
  • 25
  • 83
  • 134

5 Answers5

37

Found a solution that worked for me here.

Using setuptools==2.0.2 I did:

setuptools.setup(
    ...
    packages=setuptools.find_packages(),
    include_package_data=True,  # use MANIFEST.in during install
    ...
)
warvariuc
  • 57,116
  • 41
  • 173
  • 227
  • 8
    Nice! That feature solves almost all the problems, and I'm surprised this is the first place I've heard of it. I think this deserves to be the accepted answer instead of the `__init__.py` hack. – rspeer Sep 11 '14 at 21:00
  • 1
    This is indeed the correct answer, this solved all my issues. The ``__init__.py`` hack didn't work for me because I already placed init files everywhere in my tree! And in addition, the init trick cannot help for non-python data files. – gaborous Dec 07 '15 at 13:52
  • this works. Notes: this isn't mentioned explicitly in the answer but remove `package_data=...` in setup.py and remove the `__init__.py` files from your data subdirectories: they're not needed with this method. -- just use MANIFEST.in and set include_package_data=True. The other thing I've experienced is that data files from MANIFEST.in need to be under the python code subtrees (of at least one of the detected packages), otherwise they don't get installed. – init_js Jun 28 '19 at 01:20
  • Also note that if you intend to load the files from MANIFEST.in later on at runtime using `os.path.dirname(__file__)` path arithmetic, adding zip_safe=False might help ensure correctness (omitting zip_false will let bdist_egg infer whether it's safe via file inspection) [docs](https://setuptools.readthedocs.io/en/latest/setuptools.html#setting-the-zip-safe-flag). See [other answer](https://stackoverflow.com/a/32635882/5556676). – init_js Jun 28 '19 at 01:28
28

I personally dislike the way setuptools mixes code and data both conceptually and implementation-wise. I think that it's that implementation that is tripping you up here. For setuptools to find and use package_data it needs for the data to reside inside of a python package. A python package can be a directory but there needs to be a __init__.py file in the directory. So it looks like you need the following (empty is fine) files:

./package_fiddler/data/__init__.py
./package_fiddler/data/stylesheets/__init__.py
Toshio Kuratomi
  • 320
  • 2
  • 2
  • 2
    I had a similar problem before and it`s indeed the __init__.py that was needed. http://stackoverflow.com/questions/3760970 – ForceMagic Nov 09 '12 at 20:31
  • 2
    I agree. Something strange is that only one `__init__.py` file in the nested data files was needed to properly install the project with `tree2` in the *edit* part of my post! – Bentley4 Nov 12 '12 at 13:39
  • 7
    ``__init__.py`` is for making a directory importable as a module -- this shouldn't be necessary for data directories. See [my answer here](http://stackoverflow.com/a/32635882/1728179). – Leo Oct 01 '15 at 09:49
  • @Leo Yes, the `__init__.py` is horrible, however`package_data` is a `distutils` feature which `setuptools` ignores. Moreover, `package_data` does not work when shipping `sdists`. In general, packaging in python is horrible; it people prefer the hacky `__init__.py` solution over the inconsistent `package_data` solution, please let them. – Herbert Sep 14 '16 at 12:31
  • Still today i had that issue that package_data was not included in generated wheels package. I had to use manifest.in with the proper include's, i had to put empty `__init__.py` files in the data folders and i needed to additionally add the data folders explicitely using packages argument to setup. Else, i used ext_modules, only. Nightmare after christmas. – Wör Du Schnaffzig Jan 21 '21 at 23:22
15

The easiest way to include package data in "setup.py" is like so:

package_data = {'<package name>': ['<path to data file within package dir>']}

So in your example:

package_data = {'package_fiddler': ['data/*', 'data/stylesheets/*']}

package_data is a dictionary where the keys are the names of the packages included in the installer. The values under these keys should be lists of specific file paths or globs/wildcards within the package directory.

You also need to include the flag:

zip_safe=False

in setup(...) if you want to be able to resolve file system paths to your data. Otherwise you can use pkg_resources to do this: http://peak.telecommunity.com/DevCenter/PythonEggs#accessing-package-resources

You definitely don't need an __init__.py file in the "data" directory - this directory is not a module and is not meant to be imported.

Leo
  • 1,077
  • 11
  • 24
  • 3
    Yes, this works. But you have to duplicate the paths in `MANIFEST.in` and `setup.py` and keep them in sync. – warvariuc Dec 07 '15 at 15:12
  • 1
    This, this, this! It's the worst of it all that the docs make one believe that MANIFEST.in is **NOT** needed, while I have not found a way to make it work without it. :( – K.-Michael Aye Feb 01 '16 at 01:56
  • This does work and was the most useful answer, what tripped me up was not realizing that the path is relative to the package folder, not the parent folder where `setup.py` resides. The example in the answer demonstrates this well enough, but take particular note of that detail if this is still not working for you. – David Parks Apr 27 '23 at 16:37
-2

use 

package_data={"data": ['package_fiddler/data',]}

instead of

packages=['package_fiddler',]
Chris
  • 1,613
  • 1
  • 18
  • 27
  • It did not change anything. When I pip install the project after repackaging it there is still nothing to be found on the virtual environment. – Bentley4 Nov 09 '12 at 11:50
  • Did you use `from setuptools import setup` or `from distutils.core import setup` in your `setup.py`? – Bentley4 Nov 09 '12 at 12:11
-2

This works for me. Hope it helps. 

package_data={
    "package_fiddler": [
        '\*.\*',
        '\*/\*.\*',
        '\*/\*/\*.\*',
    ],
},
Khalil M
  • 1,788
  • 2
  • 22
  • 36
Andres Ulloa
  • 39
  • 1