How can I make pip install package data (a config file)?

Question

I have a package called clana (Github, PyPI) with the following structure:

.
├── clana
│   ├── cli.py
│   ├── config.yaml
│   ├── __init__.py
│   ├── utils.py
│   └── visualize_predictions.py
├── docs/
├── setup.cfg
├── setup.py
├── tests/
└── tox.ini

The setup.py looks like this:

from setuptools import find_packages
from setuptools import setup

requires_tests = [...]

install_requires = [...]


config = {
    "name": "clana",
    "version": "0.3.6",
    "author": "Martin Thoma",
    "author_email": "info@martin-thoma.de",
    "maintainer": "Martin Thoma",
    "maintainer_email": "info@martin-thoma.de",
    "packages": find_packages(),
    "entry_points": {"console_scripts": ["clana=clana.cli:entry_point"]},
    "install_requires": install_requires,
    "tests_require": requires_tests,
    "package_data": {"clana": ["clana/config.yaml"]},
    "include_package_data": True,
    "zip_safe": False,
}

setup(**config)

How to check that it didn't work

Quick

python3 setup.py sdist
open dist/clana-0.3.8.tar.gz  # config.yaml is not in this file

The real check

I thought this would make sure that the config.yaml is in the same directory as the cli.py when the package is installed. But when I try this:

virtualenv venv
source venv/bin/activate
pip install clana
cd venv/lib/python3.6/site-packages/clana
ls

I get:

cli.py  __init__.py  __pycache__  utils.py  visualize_predictions.py

The way I upload it to PyPI:

python3 setup.py sdist bdist_wheel && twine upload dist/*

So the config.yaml is missing. How can I make sure it is there?

did you try the path `config.yaml` instead of `clana/config.yaml` ? — Erwan, Sep 21 '19 at 07:42
related: https://stackoverflow.com/questions/29036937/how-can-i-include-package-data-without-a-manifest-in-file — Erwan, Sep 21 '19 at 07:44
It works with `MANIFEST.in` (thank you, Erwan!) - but I thought with the `include_package_data=True` I would not need that. Why is `include_package_data` not working? — Martin Thoma, Sep 21 '19 at 08:10
`sdist` is mostly controlled by the files [`MANIFEST` or its template `MANIFEST.in`](https://docs.python.org/3/distutils/sourcedist.html#manifest), not `setup.py`. `include_package_data=True` in `setup.py` is for `bdist_*` (`bdist_egg`, `bdist_wheel`, etc) commands. — phd, Sep 21 '19 at 10:54
@MartinThoma In the answer I post earlier, it seems `package_data` doens't work if `include_package_data` is true. did you try to remove `include_package_data` in your config ? — Erwan, Sep 22 '19 at 09:55

Erwan · Answer 1 · 2019-09-21T08:07:41.143

10

You can add a file name MANIFEST.in next to setup.py with a list of the file you want to add, wildcard allowed (ex: include *.yaml or include clana/config.yaml) then the option include_package_data=True will activate the manifest file

edited Sep 21 '19 at 08:07

answered Sep 21 '19 at 07:37

Erwan

587
4
23

This works, thank you! (+1) - I don't accept this answer (by now), because I hope that somebody knows why `include_package_data` did not work. – Martin Thoma Sep 21 '19 at 08:11

score 8 · Answer 2 · answered Sep 22 '19 at 15:31

In short: add config.yaml to MANIFEST.in, and set include_package_data. One without the other is not enough.

Basically it goes like this:

MANIFEST.in adds files to sdist (source distribution).
include_package_data adds these same files to bdist (built distribution), i.e. it extends the effect of MANIFEST.in to bdist.
exclude_package_data prevents files in sdist to be added to bdist, i.e. it filters the effect of include_package_data.
package_data adds files to bdist, i.e. it adds build artifacts (typically the products of custom build steps) to your bdist and has of course no effect on sdist.

So in your case, the file config.yaml is not installed, because it is not added to your bdist (built distribution). There are 2 ways to fix this depending on where the file comes from:

either the file is a build artifact (typically it is somehow created during the ./setup.py build phase), then you need to add it to package_data ;
or the file is part of your source (typically it is in your source code repository), then you need to add it to MANIFEST.in, set include_package_data, and leave it out of exclude_package_data (this seems to be your case here).

See:

@MartinThoma Did that answer the question? Anything else you would like to be clarified? — sinoroc, Jul 29 '20 at 16:29

score 0 · Answer 3 · answered Apr 06 '20 at 15:30

Following from the documentation on including data files, if your package has data files such as .yaml files, you may include them like so:

setup(
    ...
    package_data={
        "": ["*.yaml"],
    },
    ...
)

This will allow any file in your package with the file extension .yaml to be included.

How can I make pip install package data (a config file)?

How to check that it didn't work

Quick

The real check

3 Answers3

Linked