urllib response headers missing

Question

I'm using Python3 urllib to try to get a URL for an Artifact using the GitHub API.

The following curl works fine, it (correctly) shows that there is a Location header with the URL I want. There is no response content: only the headers matter:

curl -i -u"$USER:$GH_ACCESS_TOKEN" https://api.github.com/repos/$ORG/$REPO/actions/artifacts/21877858/zip

For the urllib code, auth is working and I get can back JSON responses just fine from other endpoints. But the headers I get back don't include Location.

    opener = get_opener() # bunch of boilerplate
    req = request.Request(uri)
    # same exact headers as curl sends.
    # It doesn't change anything if I change `Accept` to "application/vnd.github.v3+json"
    # like GH recommends https://docs.github.com/en/free-pro-team@latest/rest/reference/actions#download-an-artifact
    req.add_header("Accept", "*/*")
    # next line doesn't make a difference
    req.add_header("User-Agent", "curl/7.64")
    with opener.open(req) as response:
        print(response.code, response.headers["Location"])
        # prints (200, None)
        # If I stringify the headers there is a completely different set than
        # what curl shows

score 1 · Accepted Answer · answered Oct 17 '20 at 20:51

curl doesn't follow redirect by default so you end up with a 302 status code with a Location header.

urllib do follow redirect automatically though, you can use one of these solutions, for example :

import urllib.request

githubToken = "YOUR_GITHUB_TOKEN"
url = "https://api.github.com/repos/OWNER/REPO/actions/artifacts/22063356/zip"

class NoRedirection(urllib.request.HTTPErrorProcessor):
    def http_response(self, request, response):
        return response
    https_response = http_response

opener = urllib.request.build_opener(NoRedirection)

req = urllib.request.Request(url, None, headers = {
    'Authorization' :f'Token {githubToken}'
})
with opener.open(req) as response:
    print(response.code, response.headers["Location"])

Output

302 https://pipelines.actions.githubusercontent.com/NSAmkKgDUSbnHdte1rYFxmKxUVJuvcQLNt6gV7000UAFVCxMSK/_apis/pipelines/1/runs/1/signedartifactscontent?artifactName=my-artifact&urlExpires=2020-10-17T20%3A49%3A00.1948907Z&urlSigningMethod=HMACV1&urlSignature=Nq0YuQKd%2FP4jzyzGklELjfzDtBO04c7HsMgJ%2B1%2Bu%2FWY%3D

urllib response headers missing

1 Answers1