0

I have a project that includes c++ binaries and python scripts, it's setup such that it should be installed using setuptools. One of the python files is intended to be both used as a script " python3 script_name.py params and for it's primary function to be used in other python projects from script_name import function. The primary function calls a binary which is in a known relative location before the installation (the user is expected to call pip install project_folder). So in order to call the binary I want to get this files location (pre installation)

To get this I used something like

Path(__file__).resolve().parent

however, since the installation moves the file to another folder like ~/.local/... this doesn't work when imported after the installation.

Is there a way to get the original file path, or to make the installation save that path somewhere?

EDIT: After @sinoroc 's suggestion I tried including the binary as a resource by putting an __init__.py in the build folder and putting

from importlib.resources import files
import build

binary = files(build).joinpath("binary")

in the main init. After that package.binary still gives me a path to my .local/lib and binary.is_file() still returns False

from importlib_resources import files

GenerateHistograms = files("build").joinpath("GenerateHistograms")

gave the same result

Chalky
  • 13
  • 5
  • can you create a symlink instead of copying the file? – Marat Nov 25 '22 at 20:12
  • I honestly don't know, I'm not super familiar with setuptools installation – Chalky Nov 26 '22 at 09:23
  • Make the binary file part of the installable package, see "package data". Use [`importlib.resources`](https://docs.python.org/3/library/importlib.resources.html) to locate package data. – sinoroc Nov 26 '22 at 10:45
  • @sinoroc Thanks for the suggestion! I Think I'm not understanding the docs right. I put an \_\_init\_\_.py in the build folder and the main directory, added the main directory as a second package and put the following in the main \_\_init\_\_.py: `from importlib.resources import files` `import build` `binary = files(build).joinpath("binary")` After that import main_package.binary still gives a path to my `.local/lib` Even if this worked, I'd be a bit uncomfortable putting an init in the build folder, I know some people like to nuke it every now and again – Chalky Nov 26 '22 at 11:37
  • Read [this](https://stackoverflow.com/a/58941536) for all the ways to read package data. – sinoroc Nov 26 '22 at 14:37

2 Answers2

1

Since you are installing your package, you also need to include your C++ binary in the installation. You cannot have a mixed setup. I suggest something like this.

In your setup.py:

from setuptools import setup, find_packages

setup(
    name="mypkg",
    packages=find_packages(exclude=["tests"]),
    package_data={
        "mypkg": [
            "binary",  # relative path to your package directory
        ]
    },
    include_package_data=True,
)

Then in your module use pkg_resources:

from pathlib import Path

from pkg_resources import resource_filename

# "binary" is whatever relative path you put in package_data
path_to_binary = Path(resource_filename("mypkg", "binary"))

pkg_resources should be pulled in by setuptools.

EDIT: the recipe above might be a bit out of date; as @sinoroc suggests, using importlib.resources instead of pkg_resources is probably the modern equivalent.

suvayu
  • 4,271
  • 2
  • 29
  • 35
  • Thanks for the answer! Yeah I struggled a bit, not realizing that both the setuptools and importlib_resources docs things needed to be done.... Do you know if there is a way to go up relative to the package name for data inclusion? The build folder is currently located next to the "python" directory and I really kind of hate the approach of having an init in the build folder – Chalky Nov 26 '22 at 12:36
  • 1
    @Chalky The "package data" should be part of a Python importable package. No, you can not read files in parent directories of the top-level import package. Probably what you are missing now is a way to move your binaries from some "build" directory into a sub-directory of your Python code (this should happen before the sdist or wheel is built). Maybe it would be easier to help you if you edited your question to show (the relevant parts of) the project directory structure. – sinoroc Nov 26 '22 at 14:42
  • @sinoroc I ended up solving it my making a second package from the root directory shared by python and build, edited the cmake stuff to create the setup.py with the correct path to the binaries folder and then used that package as an anker for the files and added the files to their data. That worked. Admittedly I think your solution of having cmake copy the binaries inside the python directory might have been a lot more elegant as there would only be one package – Chalky Nov 26 '22 at 17:28
0

So I managed to solve it in with @sinroc 's approach in the end

In setup.py

package_data={'package':['build/*']}
include_package_data=True

and in the primary __init.py__:

from importlib.resources import files
binary = files("package.build").joinpath("binary")

So I could then from package import binary to get the path

EDIT: Looks like someone also pointed the error in my ways out before I finished ^^

Chalky
  • 13
  • 5