pulling a git repo at a particular commit in python setup.py file

Question

I have a Python project where I am using the maskrcnn_benchmark project from facebook research.

In my continuous integration script, I create a virtual environment where I install this project with thee following steps:

 - git clone https://github.com/facebookresearch/maskrcnn-benchmark.git
- cd maskrcnn-benchmark
- git reset --hard 5ec0b91cc85163ac3b58265b3f9b39bb327d0ba6
- python setup.py build develop

This works fine and installs everything in the virtual environment as it needs to be.

Now I have a setup.py for my project for packaging and deploying my app. How can I do the same in this setup.py file i.e. pull and build this repository from the particular commit hash?

Thanks to the answer below and the comments, I have the setup.py as follows now:

install_requires=[
        '5ec0b91cc85163ac3b58265b3f9b39bb327d0ba6-0.1',
        'ninja',
        'yacs',
        'matplotlib',
        'cython==0.28.5',
        'pymongo==3.7.1',
        'scipy==1.1.0',
        'torch==1.0.0',
        'torchvision==0.2.1',
        'opencv_python==3.4.2.17',
        'numpy==1.15.1',
        'gputil==1.3.0',
        'scikit_learn==0.19.2',
        'scikit_image==0.14.0',
        'sk_video==1.1.10'
  ],

dependency_links=[
        'http://github.com/facebookresearch/maskrcnn-benchmark/tarball/master#egg=5ec0b91cc85163ac3b58265b3f9b39bb327d0ba6-0.1'
  ],

No matter where I put the '5ec0b91cc85163ac3b58265b3f9b39bb327d0ba6-0.1', the maskrcnn-benchmark project gets compiled first. How can I do it that the dependency and this package is installed last?

Can't use `dependency_links=[ "git+ssh://git@xxxxx.git" ]` ? — IMParasharG, Mar 13 '19 at 16:29
@GovindParashar I see this but can I pull the project at the specific commit? The `git reset --hard` step? — Luca, Mar 13 '19 at 16:37
you can specify commit hash in the url i.e. master#egg=XXXXXXXXXXX — IMParasharG, Mar 13 '19 at 16:40
@GovindParashar Thank you. Let me try and update the thread here. Please feel free to write as an answer if you wish — Luca, Mar 13 '19 at 16:41
for i.e. https://github.com/GovindParashar136/spring-boot-web-jsp/tarball/master#egg=8138cc3fd4e11bde31e9343c16c60ea539f687d9 — IMParasharG, Mar 13 '19 at 16:44
@GovindParashar tried your suggestion but it seems that the downloaded tarball never gets installed, even though it has a setup.py file — Luca, Mar 13 '19 at 17:44

score 4 · Accepted Answer · answered Mar 13 '19 at 16:46

4

You can use dependency_links setup.py

i.e.

dependency_links =[https://github.com/GovindParashar136/spring-boot-web-jsp/tarball/master#egg=8138cc3fd4e11bde31e9343c16c60ea539f687d9]

In your case url

https://github.com/facebookresearch/maskrcnn-benchmark/tarball/master#egg=5ec0b91cc85163ac3b58265b3f9b39bb327d0ba6

answered Mar 13 '19 at 16:46

IMParasharG

1,869
1
15
26

So, I tried the following in the dependency links: `https://github.com/facebookresearch/maskrcnn-benchmark/tarball/master#egg=5ec0b91cc85163ac3b58265b3f9b39bb327d0ba6-0.1'. The project has a `setup.py` file. I can see it gets downloaded but it never gets installed. Do I need to do something else to get it to install it? I added the `0.1` as I saw that it is needed to add the version number. – Luca Mar 13 '19 at 17:19
you have to specify --process-dependency-links while using pip – IMParasharG Mar 13 '19 at 18:05
I have seen dependency links processing has been deprecated. – IMParasharG Mar 13 '19 at 18:08
but I am invoking it as `python setup.py build install`. So not using pip directly from the command line – Luca Mar 13 '19 at 18:10
Then you have add `install_requires=[ '5ec0b91cc85163ac3b58265b3f9b39bb327d0ba6-0.1' ],` before dependency_links – IMParasharG Mar 13 '19 at 18:17
Thank you. At least that tries to compile the thing. So, is it possible to change the order of installation. It seems it processes the dependency first now and the install order gets messed up. I have edited my question to reflect this. – Luca Mar 13 '19 at 18:29
If your issue resolved then please accept my answer – IMParasharG Mar 13 '19 at 18:55

score 2 · Answer 2 · answered Jul 20 '20 at 13:11

This answer suggests that including a package@ prefix to the git url will install the git commit specified:

# in setup.py
setup(
    # other fields
    install_requires=[
        "packagename@git+https://github.com/<user>/<repo>#<commit hash>",
    ],
)

so in your case:

# in setup.py
setup(
    # other fields
    install_requires=[
        "maskrcnn_benchmark@git+https://github.com/facebookresearch/maskrcnn-benchmark.git#5ec0b91cc85163ac3b58265b3f9b39bb327d0ba6",
    ],
)

This doesn't work if you want to target a commit for a package that has an entry in pypi — Keiran Raine, Aug 06 '21 at 10:35

pulling a git repo at a particular commit in python setup.py file

2 Answers2