Any python function to get "data_files" root directory?

Question

This should be a very common question for developers who used "setup.py" to build installation packages and it should be asked before but I couldn't find the proper answer anywhere.

In setup.py

from distutils.core import setup
setup(
    ....,
    ....,
    data_files=[('MyApp/CBV', ['myapp/data/CBV/training.cbv', 'myapp/data/CBV/test.cbv'])],
    ....,
    ....,
    )

Result of sudo python setup.py install

running install
running build
running build_py
running build_scripts
running install_lib
running install_scripts
changing mode of /usr/local/bin/MyApp_trainer to 755
changing mode of /usr/local/bin/MyApp_reference_updater to 755
changing mode of /usr/local/bin/MyApp_predictor to 755
changing mode of /usr/local/bin/reference_updater to 755
running install_data
creating /usr/local/MyApp/CBV
copying MyApp/data/CBV/training.cbv -> /usr/local/MyApp/CBV
copying MyApp/data/CBV/test.cbv -> /usr/local/MyApp/CBV
running install_egg_info
Removing /usr/local/lib/python2.7/dist-packages/MyApp-0.1.0.egg-info
Writing /usr/local/lib/python2.7/dist-packages/MyApp-0.1.0.egg-info

From the observation using the result above, "/usr/local" is the "data_files" root directory. Other than hardcoding, are there any Python functions that can give me this "data_files" root directory?

Minimal runnable published working example at: https://stackoverflow.com/questions/3596979/manifest-in-ignored-on-python-setup-py-install-no-data-files-installed/60735402#60735402 — Ciro Santilli OurBigBook.com, Mar 18 '20 at 07:53

Martijn Pieters · Accepted Answer · 2021-03-30T10:58:49.337

21

By default, when installing a package as root, relative directory names in the data_files list are are resolved against either the value of sys.prefix (for pure-python libraries) or sys.exec_prefix (for libraries with a compiled extension), so you can retrieve your files based on that. Qouting from the distutils documentation:

If directory is a relative path, it is interpreted relative to the installation prefix (Python’s sys.prefix for pure-Python packages, sys.exec_prefix for packages that contain extension modules).

So for your example, you'll find your files in os.path.join(sys.prefix, 'MyApp', 'CBV').

However, you would be better off using the importlib.resources library (Python 3.7 and up) to load package data. You do want your data files included in the package for that to work best. That means you would not use data_files but instead either list file patterns in a MANIFEST.in file and set include_package_data=True, or list file patterns in package_data, see Including data files in the setuptools documentation.

For earlier Python versions, you can do the same with the pkg_resources module Resource API to load data files (it is part of the setuptools library, for this very purpose).

You can then load such resource files straight from the package into a string with resource_string() for example:

try:
    from importlib.resources import read_text
except ImportError:
    from pkg_resources import resource_string as read_text

foo_config = read_text(__name__, 'foo.conf')

edited Mar 30 '21 at 10:58

answered Jan 08 '13 at 09:14

Martijn Pieters

1,048,767
296
4,058
3,343

Currently, I'm using "setup" from "distutils". It may take a while to test if your solution can help me :) – Jessada Thutkawkorapin Jan 08 '13 at 09:37
@JessadaThutkawkorapin: Switching to `setuptools` is easy enough. :-) It's backwards compatible to `distutils`, and definitely recommendable! – Martijn Pieters Jan 08 '13 at 09:39
While I was looking for the difference between disutils and setuptools, I found http://stackoverflow.com/questions/6344076/differences-between-distribute-distutils-and-setuptools. From what I roughly read, it seems like the suggestion was going toward disutils2. Do you still recommend setuptools? – Jessada Thutkawkorapin Jan 08 '13 at 09:59
I still recommend `distribute` (the `setuptools` fork). `distutils2` has yet to materialize. `distribute` is the current defacto standard for python package distribution. – Martijn Pieters Jan 08 '13 at 10:00
2

This `resource_string` doesn't appear to work for `data_files` that specify that the file should appear at root or some subdirectory outside of the package directories. – CMCDragonkai Jan 30 '19 at 01:17
@CMCDragonkai: no, the Resource API (including `resource_string()`) only works with `include_package_data` / `package_data` files, **not** with `data_files` locations. That's why my answer says *You do want your data files included in the package for that to work best* followed by a link on how to do that. – Martijn Pieters Jan 30 '19 at 11:59
sys.prefix doesn't work for me. In a venv sys.prefix gives the venv root which corresponds to the data_files root. Outside venv with the pip lauched as sudo, sys.prefix gives /usr and the data_files root is /usr/local ! – michael.t May 20 '19 at 13:43
@michael.t: a virtualenv changes the location where stuff gets installed, yes. That’s the purpose of a virtualenv. What you describe is expected behaviour. – Martijn Pieters May 20 '19 at 23:20
3

@Martijn Pieters This is obviously OK. The problem is that in a venv `data_files` gets installed in `sys.prefix` and outside a venv it gets installed in `sys.prefix/local` Considering that the app doesn't know how it has been installed, it cannot know if the `data_files` end up in `sys.prefix` or `sys.prefix/local`. – michael.t May 21 '19 at 08:41
@michael.t: I'd have to go deep-dive again why that would be; IIRC I explicitly traced the distutils / setuptools intallation code to determine how `sys.prefix` is being used, and virtualenv basically sets `sys.prefix`. I don't have much time at the moment however :-/ – Martijn Pieters May 21 '19 at 09:54
Great answer, but what about the [editable installation](https://pip.pypa.io/en/stable/reference/pip_install/#install-editable) – betontalpfa Mar 30 '21 at 10:23
@betontalpfa: this works just the same with an editable installation. – Martijn Pieters Mar 30 '21 at 11:15
I mean, the `data_files`, the `sys.prefix` pattern wont work with `data_files` in editable mode – betontalpfa Mar 30 '21 at 11:23
@betontalpfa: sorry, I see what you mean. Which is why you want to use `importlib.resources` and `package_data`, which works when using editable mode. – Martijn Pieters Mar 30 '21 at 12:11
@betontalpfa: the alternative being that you build in a fallback to a relative path to your module to handle the editable case. – Martijn Pieters Mar 30 '21 at 12:11

Any python function to get "data_files" root directory?

1 Answers1

Linked