4

We have a private repo containing a Python project that is being built by setuptools on a CI/CD server and then the wheel artifact is being pushed back to Github as a release. This process works great, however getting the resulting wheel back into other build processes that require this as a dependancy does not seem so easy.

Our original process was using git submodules, however the wheel format is much better self contained, and the consuming project does not have to worry about relative paths, because the wheel is installed straight into site-packages.

The biggest issue has come with pulling and installing the wheel in any consuming projects. Adding the url of the wheel to the requirements.txt gets pip to try to pull the wheel, but it then fails with the following:

Could not install requirement <ProjectName>==2.0 from
https://github.com/<CompanyName>/<ProjectName>/releases/download/v2.0.0/<ProjectName>-2.0-py3-none-any.whl (from -r 
requirements.txt (line 27)) because of HTTP error 404 Client Error: Not Found for url

A 404 error is also returned if following the url in a private browser window. If the url is followed on a browser already logged in to GitHub, the wheel is returned. Inspecting the request that got the wheel shows that the browser sent a cookie that let GitHub know that this wheel belonged to me, but sending cookies programatically doesn't seem like a great idea.

The following questions both touch on the same topic. The accepted answer for the first question seems not very elegent - hopefully there is a better method somewhere!

How do I download binary files of a GitHub release?

pip install wheel version from private github repo

Thanks

Stuart

Stuart Buckingham
  • 1,574
  • 16
  • 25
  • 1
    You'll have to authenticate the pull. – dan1st Jan 21 '20 at 19:05
  • Thanks @dan1st. It looks like the only way pip allows authentication is from user:pass (https://github.com/pypa/pip/issues/51). Also looked at the pip source code to see if there was a back door for token authentication, but it seems like there isn't. – Stuart Buckingham Jan 21 '20 at 19:24
  • 1
    You can use a PAT instead of a password in any case. – dan1st Jan 21 '20 at 19:25
  • Hmmmm, I tried even just with curl and was unsuccessful: `curl -H "Authorization: token ####" https://github.com///releases/download/v2.0.0/-2.0-py3-none-any.whl` This is not going through the API, but rather directly to the artifacts download link – Stuart Buckingham Jan 21 '20 at 19:34
  • You can use the token just like the username, e.g. `username:token@github.com/org/repo` – dan1st Jan 21 '20 at 19:35
  • I think you can also login on the web ui if the token includes the required permissions. – dan1st Jan 21 '20 at 19:36
  • Still 404ing. Does that work for pulling release artifacts or just git repos? – Stuart Buckingham Jan 21 '20 at 20:11
  • Depends on the permissions of the PAT, I think – dan1st Jan 21 '20 at 20:12
  • It should work with the permission `repo:private` – dan1st Jan 21 '20 at 20:24
  • 1
    I have this issue too and nothing so far suggested works. Anyone with new ideas? I've created a PAT, but don't see an option for `repo:private` perms, only for public. Using the PAT, I _can_ convince pip to download the source code, but _can't_ get it to dl a wheel. I've tested the wheel URL to be correct in browser. This works: `pip install git+https://${GITHUB_PAT}@github.com//.git@v0.1.0` But this doesn't work: `pip install https://${GITHUB_PAT}@github.com///releases/download/v0.1.0/-0.1.0-py2.py3-none-any.whl` – jonsedar Feb 12 '21 at 08:48

1 Answers1

2

There's a convenient way to authenticating the pull using a GitHub access token that will act pretty transparently across quite a few different tools that speak HTTP. This is the ~/.netrc file on most systems. Try the following:

$ touch ~/.netrc && chmod 600 ~/.netrc && cat >> ~/.netrc << 'EOF'
machine github.com
login your_githubu_sername
password your_github_api_token
EOF

Last I checked, curl, wget and git (among others) will respect this file by default for authenticating HTTP(S) requests. Though I'm not sure that this is the case in every configuration, you may be surprised at how ubiquitous the support for ~/.netrc is across different tools. This is a Unixsm, I have no idea if it is supported on MacOS or Windows, but most Linux distributions have support for it in their tools

adam
  • 384
  • 2
  • 9