Python Packaging with .pth files

Question

I have a suite of packages that are developed together and bundled into one distribution package.

For sake of argument, let's assume I have Good Reasons for organizing my python distribution package in the following way:

SpanishInqProject/
|---SpanishInq/
|     |- weapons/
|     |   |- __init__.py
|     |   |- fear.py
|     |   |- surprise.py    
|     |- expectations/
|     |   |- __init__.py
|     |   |- noone.py
|     |- characters/
|         |- __init__.py
|         |- biggles.py
|         |- cardinal.py
|- tests/
|- setup.py
|- spanish_inq.pth

I've added the path configuration file spanish_inq.pth to add SpanishInq to the sys.path, so I can import weapons, .etc directly.

I want to be able to use setuptools to build wheels and have pip install weapons, expectations and characters inside the SpanishInq directory, but without making SpanishInq a package or namespace.

My setup.py:

  from setuptools import setup, find_packages

  setup(
    name='spanish_inq',
    packages=find_packages(),
    include_package_data=True,       
   )

With a MANIFEST.in file containing:

   spanish_inq.pth

This has been challenging in a couple of ways:

pip install has put weapons etc. directly in the site-packages directory, rather than in a SpanishInq dir.
my spanish_inq.pth file ends up in the sys.exec_prefix dir, rather than in my site-packages dir, meaning the relative path in it is now useless.

The first problem I was able to sort of solve by turning SpanishInq into a module (which I'm not happy about), but I still want to be able to import weapons etc. without SpanishInq as a namespace, and to do this I need SpanishInq added to the sys.path, which is where I was hoping the .pth file would help...but I can't get it to go where it ought to.

So...

How do I get the .pth file to install into the site-packages dir?

You want to be able to write `import weapons` *and* for `weapons/` to be stored inside a `SpanishInq` directory when installed? Why? What possible purpose does `SpanishInq` serve? — jwodder, Jun 25 '18 at 23:05
Merely as a container. Let's assume there's potentially dozens and dozens of packages in SpanishInq, and rather than clutter up site-packages, we want them nicely grouped together, since they all install as a bundle anyways. To make things more interesting, imagine that we have several projects (each working in different environments) where each project may fork SpanishInqProject and each fork may contain varying numbers of packages. Let's just say I have certain constraints I need to work with. I have Good Reasons. — Ryan de Kleer, Jun 25 '18 at 23:31
...though I am interested in hearing the drawbacks or caveats of grouping the packages in this way. — Ryan de Kleer, Jun 25 '18 at 23:47

score 1 · Answer 1 · answered Feb 16 '22 at 07:30

This is very similar to setup.py: installing just a pth file? (this question is strictly a superset, in terms of functionality) -- I've adapted the relevant part of my answer there below.

The right thing to do here is to extend setuptools' build_py, and copy the pth file into the directory into the build directory, in the location where setuptools prepares all the files that go into site-packages there.

from setuptools.commands import build_py


class build_py_with_pth_file(build_py):
     """Include the .pth file for this project, in the generated wheel."""

     def run(self):
         super().run()

         destination_in_wheel = "spanish_inq.pth"
         location_in_source_tree = "spanish_inq.pth"
 
         outfile = os.path.join(self.build_lib, destination_in_wheel)
         self.copy_file(location_in_source_tree, outfile, preserve_mode=0)

setup(
   ...,
   cmdclass={"build_py": build_py_with_pth_file},
)

Python Packaging with .pth files

1 Answers1