Using git submodules with python

Question

I've read a lot of blog posts and questions on this site about the usage of git submodules and still have no idea how to better use them with python.

I mean, what is the easier way to manage dependencies if I have such a package:

├── mypkg
│   └── __init__.py
├── setup.py
└── submodules
    ├── subm1
    └── subm2

Then, what if I need to use "mypkg" as a submodule for "top_level_pkg":

├── setup.py
├── submodules
│   └── mypkg
└── top_level_package
    └── __init__.py

, I want to run pip install . and have all resolved correctly (have each submodule installed to the VENV in correct order).

What I've tried:

Install each submodule using "pip" running in a subprocess. But it seems to be a hacky way and hard to manage (Unexpected installation of GIT submodule)
Use "install_requires" with "setuptools.find_packages()" but without success
Use requirements.txt file for each submodule, but I can't find a way how to automate it so "pip" could automatically install all requirements for all submodules.

Ideally, I imagine a separate setup.py file for each submodule with install_requires=['submodules/subm1', 'submodules/submn'], but setuptools does not support it.

Do the submodules have their own `setup.py` or not? In other words are they projects that are _pip_ installable independently? — sinoroc, Nov 24 '20 at 14:39
Have you tried `install_requires=['subm1 @ submodules/subm1', '...']`? — sinoroc, Nov 24 '20 at 16:19
Otherwise you might not need submodules at all: `install_requires=['subm1 @ git+https://repo.dev/subm1.git', '...']`. — sinoroc, Nov 24 '20 at 16:24
@sinoroc, No, i have not even noticed such syntax in the documentation. Thank you, I'll try it. — garrywreck, Nov 24 '20 at 16:29
@sinoroc, It seems to be incorrect syntax because I get: "'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Invalid URL: submodules/subm1" — garrywreck, Nov 25 '20 at 06:55
Ah yes, I should have linked the specification:https://www.python.org/dev/peps/pep-0440/#direct-references -- you need an absolute path, something like `subm1 @ file:///path/to/subm1`. So it might not work for your use case. Try the git variant instead. You would not need to worry about git submodules then. — sinoroc, Nov 25 '20 at 09:00
It seems like the final answer is that there's no PORTABLE way to construct a python project with git submodules and have setuptools install the git submodule code before the parent, do I have that right? Using an absolute path in a file:// URL in setup.py fails on Jenkins CI build servers, just to name one glaring problem. — chrisinmtown, Jan 13 '22 at 19:43

score 1 · Answer 1 · answered Nov 24 '20 at 16:02

1

I'm not saying it's impossible, but very hard and very tricky. A safer way is to turn each submodule into an installable Python module (with it's own setup.py) and install the submodules from Git.

This link describes how to install packages from Git with setup.py: https://stackoverflow.com/a/32689886/2952185

answered Nov 24 '20 at 16:02

Gijs Wobben

1,974
1
10
13

It turned out to be not very hard and not very tricky, at least if I understood the OP's question correctly, please see my answer. – chrisinmtown Apr 22 '22 at 18:14

score 1 · Answer 2 · answered Nov 25 '20 at 13:58

1

Thankfully to Gijs Wobben and sinoroc I came up with solution that works for my case:

install_requires=['subm1 @ file://localhost/<CURENT_DIR>/path/to/subm1']

answered Nov 25 '20 at 13:58

garrywreck

63
1
6

An absolute path works but is hopelessly unportable. Like @garrywreck I want to use a relative path for portability but just get the error `'install_requires' must be a string or list of strings containing valid project/version requirement specifiers` – chrisinmtown Jan 13 '22 at 19:37

chrisinmtown · Answer 3 · 2022-04-25T17:45:29.323

I have managed to install a Python package from a git submodule together with a main package. These are proprietary and are never published to PyPI. And both pip and tox seem to work just fine.

To set the context, I have a git repo with a single Python package and a single git submodule; the git submodule also contains a single Python package. I think this structure is as generic and simple as it can possibly be, here's a visualization:

main-git-repo-name
├── mainpkg
│   └── __init__.py
├── setup.py
├── tests
└── util-git-repo-name (this is a git submodule)
    ├── setup.py
    ├── test
    └── utilpkg
        └── __init__.py

I wanted to have pip install everything in a single invocation, and the utilpkg should be usable in mainpkg via just import utilpkg (not nested oddly).

The answer for me was all in setup.py:

First, specify the packages to install and their locations:

packages=find_packages(exclude=["tests"])
       + find_packages(where="util-git-repo-name/utilpkg", exclude=["test"]),
package_dir={
    "mainpkg": "mainpkg",
    "utilpkg": "util-git-repo-name/utilpkg"
},

Second, copy all the install_requires items from the git submodule package's setup.py file into the top level. In my case the utility package is an API client generated by swagger-codegen, so I had to add:

install_requires=[
    "urllib3 >= 1.15", "six >= 1.10", "certifi", "python-dateutil",
    ...],

Anyhow, when running pip3 install . this config results in exactly what I want in the site-packages area: a directory mainpkg/ and a directory utilpkg/

HTH

Using git submodules with python

3 Answers3

Linked