0

I have such project structure:

├── mypkg
│   └── __init__.py
├── setup.py
└── submodules
    └── subm
        ├── setup.py
        └── subm
            └── __init__.py

And I need to install submodules/subm so it would be available in the VENV. I try to do it by calling "pip" in a subprocess. So I have such "setup.py" files:

mypkg/setup.py:

#!/usr/bin/env python

import inspect
import os
import sys
import subprocess
import setuptools
from setuptools.command.install import install
from setuptools.command.develop import develop


SUBMODULE_DEPENDENCIES = {
    # dependency directory: extra (if any)
    'subm': None,
}

# It will be /tmp/pip-... during deploy installation, and local dir in devel
CUR_DIR = os.path.dirname(inspect.getfile(inspect.currentframe()))


def post_install(devel=False):
    """
    Post-install script to install submodule dependencies
    """
    submodules_dir = os.path.abspath(os.path.join(CUR_DIR, 'submodules'))
    # install packages from each submodule
    for dependency in SUBMODULE_DEPENDENCIES:
        extra = SUBMODULE_DEPENDENCIES.get(dependency)
        extra = '[{}]'.format(extra) if extra else ''

        call_args = [sys.executable, '-m', 'pip', 'install', '-v',
                     os.path.join(submodules_dir, dependency, extra)]
        subprocess.run(call_args)


class LocalInstall(install):
    """
    Register post-install script to run when installing in normal mode
    """
    def run(self):
        install.run(self)
        post_install()


class LocalDevelop(develop):
    """
    Register post-install script to run when installing in development mode
    """
    def run(self):
        develop.run(self)
        post_install(devel=True)


setuptools.setup(
    name='mypkg',
    version='1.0.0',
    install_requires=['gevent'],
    packages=['mypkg'],
    cmdclass={'install': LocalInstall, 'develop': LocalDevelop},
)

subm/setup.py:

#!/usr/bin/env python
import setuptools


setuptools.setup(
    name='subm',
    version='1.0.0',
    packages=['subm'],
    install_requires=['gevent==1.4'],
)

Then I run:

pip install . && pip freeze

And get "gevent-20.9.0" installed while I expect to have "gevent-1.4". Also, I actually have both gevent versions (1.4 and 20.9.0) somehow. So I can run pip uninstall gevent twice and remove both versions.

When I run:

pip install -e . && pip freeze

I get expected version of pip (1.4).

What may be wrong here? How may I try to configure "setup.py" to correctly resolve submodule dependencies?

garrywreck
  • 63
  • 1
  • 6
  • Why are you using submodules? Why are the submodules placed in these locations if they are independently installable projcets? Why not use them as raw packages (without `setup.py`) instead? – sinoroc Nov 25 '20 at 09:33
  • @sinoroc, Unfortunately, I don't have a good answer to your questions because using submodules was not my idea. I just have to deal with it using the project structure I described here. But it seems it is not actual anymore since I've got success in https://stackoverflow.com/questions/64988110/using-git-submodules-with-python. But it leaves some gaps in understanding of packaging. – garrywreck Nov 25 '20 at 14:04
  • OK, if you can not influence the project structure, then it is probably pointless to discuss about it. All in all, it is a quite bad project structure from my point of view, but on the other hand I do not know the conditions that led to this structure. – sinoroc Nov 25 '20 at 15:19

0 Answers0