Is it possible to include subdirectories using dist utils (setup.py) as part of package data?

Question

Basically my python package is setup like:

module
\_examples
  \_folder1
     \_file1.py
     \_file2.py
  \_folder2
    \_file1.py
    \_file2.py

Basically I want to just use:

package_data  = { 
            'module': ['examples/*'],
  },

because my project always has people adding examples and I want it to be easy to list them from within my application. I can get it work for any FILE within examples, but not re-curse down through sub-directories. Is this possible?

score 53 · Answer 1 · edited Apr 09 '19 at 20:41

53

I believe what you're looking for is something like this for you setup.py, which will recursively find any packages in the project, also be sure and include __init__.py files to subdirectories for each package you want.

from setuptools import setup, find_packages

setup(name='MySoftware',
      packages=find_packages()
)

edited Apr 09 '19 at 20:41

hynekcer

14,942
6
61
99

answered Mar 16 '14 at 20:12

defermat

633
5
6

2

Be careful here, because you might accidentally include test directories, etc. unless you explicitly `exclude=...` these. – Arminius Jan 09 '18 at 23:21

score 15 · Answer 2 · edited Mar 29 '22 at 08:03

In my package root folder, I have a setup.py file see doc
In this file, I have the following code:

from setuptools import setup

with open("README.md", "r") as fh:
    long_description = fh.read()

setup(
    name='package name',
    version='0.4.1',
    description='short description',
    long_description=long_description,
    long_description_content_type="text/markdown",
    url='repository url',
    author='My name',
    author_email='my@e.mail',
    license='MIT',
    packages=['PackageName','PackageName.SubModule'],
    zip_safe=False,
    install_requires=[
        'dependecy1',
    ],
    classifiers=[
        'Development Status :: 3 - Alpha',
        'License :: OSI Approved :: MIT License',
        'Programming Language :: Python :: 3.7'
    ]
)

The interesting part to answer the question, here is : packages=['PackageName','PackageName.SubModule'],

By following this syntax, you can include sub-modules to your main package distribution.

More info about all the others arguments can be found in the doc.

Out of all the answers here, I think this is the cleanest way — Teshan Shanuka J, Aug 03 '21 at 11:02

score 7 · Answer 3 · answered Jun 07 '12 at 02:40

7

You'll have to use a MANIFEST.in file for that.

I believe you'll want something like this:

$ cat MANIFEST.in
recursive-include examples/ *.py

answered Jun 07 '12 at 02:40

David Wolever

148,955
89
346
502

I think you can also directly use Python functions in setup.py to find files and directories, but I can't quite find the exact answer at the moment. – Jeff Tratner Jun 07 '12 at 05:20
1

Hrm, I will have to give it a try as I did: recursive-include examples *.py and didn't use the /. I read that MANIFEST.in is being deprecated though. I think I may just have to follow the python function call method. – xamox Jun 07 '12 at 17:58
2

Oh… Well, if you figure out how to do it with Python, please post here. I'd like to know. – David Wolever Jun 07 '12 at 18:34
where is the location of the Manifest.in file? Thank you – vel Jul 05 '22 at 14:20

score 4 · Answer 4 · edited May 23 '17 at 12:09

Yes, you can include all the subdirectories.

You just need to pass the below args to setup() function:

packages=find_packages()

include_package_data=True

Along with this you need to have a MANIFEST.in file, with contents as

recursive-include examples *

This ensures all the files are recursively included.

If you want to exclude certain extensions specifically, you can do so by specifying exclude array in the find_packages() argument.

Ex: To exclude .txt files

packages=find_packages(exclude=['.txt'])

You can read more about include_package_data here.

And also here is another link which tells you when you should not use include_package_data

stason · Answer 5 · 2020-02-21T19:06:29.440

None of the suggested answers worked for me in a similar situation.

I needed to make a distribution with my package, which included several sub-modules in a sub-directory, so that these were the files I needed to go into sdist:

ipyexperiments/*py
ipyexperiments/utils/*py

and no matter what I tried, the subdir utils's modules were not getting included in sdist.

What worked for me is leaving config.py's default:

# config.py
from setuptools import setup, find_packages
[...]
setup(
    packages = find_packages(),
    [...]
)

but adding to MANIFEST.in:

# MANIFEST.in
graft ipyexperiments

and everything under ipyexperiments was included.

If you don't already have MANIFEST.in, create it at the same directory as config.py.

I also added to MANIFEST.in

prune tests
global-exclude *.py[co]

to exclude all of tests directory and any unwanted *pyc and *.pyo files anywhere.

where does your `MANIFEST.in` file live? At the same level as `setup.py` or at the top of the package you're making? — Mason Caiby, Feb 21 '20 at 17:04
that's correct. I edited the reply to reflect that. thank you for asking. — stason, Feb 21 '20 at 19:06

score 2 · Answer 6 · answered Mar 16 '21 at 12:31

2

You can make use of include parameter of find_packages():

...
setup(name="module",
      packages=find_packages(include=('module*',)),
...

answered Mar 16 '21 at 12:31

Jahid

21,542
10
90
108

score 0 · Answer 7 · answered Nov 15 '18 at 15:36

0

Following what David Wolever said, just to make it a little more clear. If you want to include everything under a sub-directory folder you have to explicitly specify each file in the MANIFEST.in,

recursive-include examples/ *.py *.png *.sh etc.....

It would be nice if the manifest.in would just understand examples/ and include everything but oh well.

answered Nov 15 '18 at 15:36

machinaWhisperer

1

It does support including all files below a certain path now with `graft some-dir/`. – Nick Sweeting Aug 18 '20 at 20:33

score 0 · Answer 8 · answered Jan 07 '23 at 11:22

0

All other answers seemed to complicated. This worked with version 9.3.15:

package_data  = { 
            'module': ['examples/**'],
  },

Note the ** instead of *. No __init__.py was required in any folders.

answered Jan 07 '23 at 11:22

Amar

1,340
1
8
20

score 0 · Answer 9 · answered Jul 26 '23 at 00:12

0

The simple answer that worked for me is to just include all the directories in the packages= option in the setup call:

 packages=[“examples”, “examples/_folder1”, “examples/_folder2”]

answered Jul 26 '23 at 00:12

Frank Petroski

1

Is it possible to include subdirectories using dist utils (setup.py) as part of package data?

9 Answers9

Linked