2

We have created a pip wheel file for a package which has a collection of python and perl scripts. Since python packaging will only add the python files to the wheel file what is the best way to package the perl files as well.

This is my project structure

.
|____myproject
| |____logging.ini
| |____utils.py
| |____myperlscript.pl
| |____config.py
| |____version.py
| |____scripta.py
| |____scriptb.py
| |____scriptc.py
| |______init__.py
|____test
| |____test_scripts.py
|______init__.py
|____MANIFEST.in
|____README.md
|____setup.py
|____.gitignore
|____Jenkinsfile
phd
  • 82,685
  • 13
  • 120
  • 165
joydeep bhattacharjee
  • 1,249
  • 4
  • 16
  • 42
  • Either use [package_data](https://stackoverflow.com/search?q=%5Bsetuptools%5D+package_data) or [data files](https://stackoverflow.com/search?q=%5Bsetuptools%5D+data+files). – phd Feb 04 '19 at 14:12

1 Answers1

0

If you are currently using setuptools in the setup.py for your project, and making use of python setup.py bdist_wheel as the method to generate the .whl file, add the following line to the MANIFEST.in file which is already present at the root of your project.

recursive-include myproject *

Naturally, replace myproject for the actual top level directory that will include the target .pl scripts (or any other file).

As a demo, if your setup.py is written approximately like so:

from setuptools import setup
from setuptools import find_packages

setup(
    name='myproject',
    version='0.0.0',
    description='demo package',
    long_description=open('README.md').read(),
    classifiers=[
        'Programming Language :: Python',
    ],
    packages=find_packages(),
    include_package_data=True,
    zip_safe=False,
)

Running python setup.py bdist_wheel will show output that looks like:

...
adding 'myproject/__init__.py'
adding 'myproject/config.py'
adding 'myproject/logging.ini'
adding 'myproject/myperlscript.pl'
adding 'myproject/scripta.py'
adding 'myproject/utils.py'
adding 'myproject/version.py'
adding 'test/__init__.py'
...

The files are packaged inside the .whl:

$ unzip -t dist/myproject-0.0.0-py3-none-any.whl 
Archive:  dist/myproject-0.0.0-py3-none-any.whl
    testing: myproject/__init__.py    OK
    testing: myproject/config.py      OK
    testing: myproject/logging.ini    OK
    testing: myproject/myperlscript.pl   OK
...

Installing the resulting .whl file in a new environment:

$ pip install -U myproject-0.0.0-py3-none-any.whl 
Processing myproject-0.0.0-py3-none-any.whl
Installing collected packages: myproject
Successfully installed myproject-0.0.0
$ ls env/lib/python3.6/site-packages/myproject/
config.py    logging.ini      __pycache__  utils.py
__init__.py  myperlscript.pl  scripta.py   version.py

Also note that if the MANIFEST.in method is unwanted, include package_data={'': ['*']}, argument for the setup call should also make it work with the recent versions of setuptools.

Further addendum: The setuptools package actually has a MANIFEST.in that include this particular syntax, though restricted to specific filename extensions for the files they want to include. This is clearly a supported option despite some guides/documentation that might suggest otherwise. In fact, this is a feature provided by the core distutils module that is shipped with Python by default. Related questions:

metatoaster
  • 17,419
  • 5
  • 55
  • 66
  • 1
    [Note: MANIFEST.in does not affect binary distributions such as wheels.](https://packaging.python.org/guides/distributing-packages-using-setuptools/#manifest-in) – phd Feb 05 '19 at 15:28
  • @phd the documentation is wrong, it does actually work, as I have projects that are packaged in this manner. – metatoaster Feb 06 '19 at 01:30
  • @phd also if you read my updates to this answer, you may come across [this](http://blog.codekills.net/2011/07/15/lies,-more-lies-and-python-packaging-documentation-on--package_data-/) link (via [this answer](https://stackoverflow.com/a/14159430/2904896)) explaining why your suggestion to use `package_data` doesn't actually work, and `data_files` take a lot more effort to use. – metatoaster Feb 06 '19 at 07:10
  • Both links point to quite an outdated answers ("damn lies" in their own terms). These days the primary distribution format is wheels, and `package_data` works perfectly. – phd Feb 06 '19 at 10:59
  • As for `MANIFEST.in` — your update answer contains `include_package_data=True,` — that makes it work. – phd Feb 06 '19 at 11:01
  • @phd Actually it still needs one more line in order to actually make it work, which is `package_data={'': ['*']},` for the new system, but it does **not** change the fact that the assertion that `MANIFEST.in` does not affect wheels is also wrong. – metatoaster Feb 06 '19 at 11:48
  • Without `include_package_data` it doesn't. – phd Feb 06 '19 at 11:50