include extra file in a Python package using setuptools

Question

I am attempting to build a python wheel using setuptools. The package needs to include two files:

• mymodule.py - a python module in the same directory as setup.py
• myjar.jar - a java .jar file that exists outside of my package directory

I am building my package using python3 setup.py bdist_wheel.

If I call setup() like so:

setup(
    name="mypkg",
    py_modules=["mymodule"],
    data_files=[('jars', ['../target/scala-2.11/myjar.jar'])]
)

then myjar.jar does successfully get included in the .whl (good so far) however when I pip install mypkg it places the jar at /usr/local/myjar.jar (this kinda explains why) which isn't what I want at all, I want it to exist in the same place as mymodule.py, i.e. /usr/local/lib/python3.7/site-packages/

If I change setup.py to

setup(
    name="mypkg",
    py_modules=["mymodule"],
    package_data={'jars': '../target/scala-2.11/myjar.jar'}
)

or

setup(
    name="mypkg",
    py_modules=["mymodule"],
    package_data={'jars': ['../target/scala-2.11/myjar.jar']}
)

then myjar.jar simply doesn't get included in the .whl. I tried copying myjar.jar into the same directory and changing setup.py to:

setup(
    name="mypkg",
    py_modules=["mymodule"],
    package_data={'jars': 'myjar.jar'}
)

or

setup(
    name="mypkg",
    py_modules=["mymodule"],
    package_data={'jars': ['myjar.jar']}
)

but still myjar.jar does not get included in the .whl.

I've been tearing my hair out over this for hours, hence why I'm here.

I've read a myriad of SO posts on this:

• How to include package data with setuptools/distribute?
• MANIFEST.in ignored on "python setup.py install" - no data files installed?
• How do you add additional files to a wheel?
• setuptools: adding additional files outside package

which suggest different combinations of data_files, package_data, include_package_data=True and/or use of a Manifest.in file but still I can't get this working as I would like, so I'm here hoping someone can advise what I'm doing wrong.

Answer 1

The data files (in that case myjar.jar) should really be package data files, and as such they should be part of a Python package. So having such files in parent directories makes things much more complicated, but probably not impossible. So let's start with a simpler example. I believe something like the following should work...

Project directory structure:

MyProject
├ MANIFEST.in
├ mymodule.py
├ setup.py
└ myjars
    ├ __init__.py
    └ myjar.jar

MANIFEST.in:

recursive-include myjars *.jar

setup.py:

#!/usr/bin/env python3

import setuptools

setuptools.setup(
    name='MyProject',
    version='1.2.3',
    #
    include_package_data=True,
    packages=['myjars'],
    py_modules=["mymodule"],
)

myjars/__init__.py might not be strictly necessary, but I believe it's better to have it. And as always, an empty __init__.py file is perfectly good enough.

(This assumes the myjars/myjar.jar file exists before the source distribution sdist is built.)

As to dealing with the data files in parent directories, my recommendation would be to simply copy (or symlink) those files before calling setup.py, maybe as part of a shell script or anything like that. There are probably ways to do the copy as part of a custom setuptools command in setup.py, but it's not worth the effort in my opinion, and really it's not part of setup.py's job.