Curl works, requests doesn't

Question

I've been told that using curl is bad and you should never find a python way to do it. Requests is king. My question is, how can I find a way to do the following command in Requests because I'm stumped.

Curl works perfectly in git bash. Problem is when I want to use python... Requests is not working for me. See below

curl -u "USER:PASS" -L -O https://api.bitbucket.org/2.0/repositories/EEEE/AAA/downloads/zip.zip

But how can I mimic this with requests. Can this be done?

import requests
from requests.auth import HTTPBasicAuth

url = "https://bitbucket.org/SYA/Music/downloads/SSR131.zip"
src = r"C:\1\1\SR131.zip"

headers = {'Content-Type': 'application/json'}

respo = requests.get(url, stream=True, verify=False, auth=('##############', '######################'), headers=headers)

if respo.status_code == requests.codes.ok:
    out = open(src, "wb")
    for block in respo.iter_content(1024):
        if not block:
            break

        out.write(block)
    out.close()
else:
    print("Not able to download ZIP url {url}: {status}, {content}".format(url=url, status=respo.status_code, content=respo.content))

Error:

Not able to download ZIP url https......: 401, b''

More specifically:

Debug output of Requests:

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.bitbucket.org
C:\Users\Imapotatofish\AppData\Local\Continuum\miniconda3\lib\site-packages\urllib3\connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
send: b'GET /2.0/repositories/Imapotatofish/EEE/downloads/zip.zip HTTP/1.1\r\nHost: api.bitbucket.org\r\nUser-Agent: python-requests/2.18.4\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nAuthorization: Basic VHlzb25kb2dlcnpAZ21haWwuY29tOlJLRDJSUlRXMnhVNnhjWGh6Y3lY\r\n\r\n'
  InsecureRequestWarning)    repositories/EEEE/AAA/downloads/zip.zi
DEBUG:urllib3.connectionpool:https://api.bitbucket.org:443 "GET /2.0/repositories/Imapotatofish/EEE/downloads/zip HTTP/1.1" 401 0
reply: 'HTTP/1.1 401 Unauthorized\r\n'
header: Server header: WWW-Authenticate header: Content-Type header: Strict-Transport-Security header: Date header: X-Served-By header: X-Static-Version header: ETag header: X-Render-Time header: Connection header: X-Version header: X-Request-Count header: X-Frame-Options header: Content-Length Not able to download ZIP url https://api.bitbucket.org/2.0/repositories/Imapotatofish/EEE/downloads/zip: 401, b''

Debug of git bash curl

curl -v -I -H "Testing: Test header so you see this works" -u "USER:PASS" -L -O https://api.bitbucket.org/2.0....zip.zip

* Connected to api.bitbucket.org (2401:1d80:1010::153) port 443 (#0)
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [100 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2417 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [149 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Atlassian, Inc.; OU=Bitbucket; CN=*.bitbucket.org
*  start date: Jun 12 00:00:00 2017 GMT
*  expire date: Jun 16 12:00:00 2020 GMT
*  subjectAltName: host "api.bitbucket.org" matched cert's "*.bitbucket.org"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Server auth using Basic with user 'Imapotatofish'
* Using Stream ID: 1 (easy handle 0x267f970)
} [5 bytes data]
> HEAD /2.0/repositories/Imapotatofish/EEE/downloads/zip.zip HTTP/2
> Host: api.bitbucket.org
> Authorization: Basic VHlzb25kb2dlcno6UktEMlJSVFcyeFU2eGNYaHpjeVg=
> User-Agent: curl/7.57.0
> Accept: */*
> Testing: Test header so you see this works
>
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
} [5 bytes data]
< HTTP/2 302
< server: nginx
< vary: Authorization
< content-type: text/html; charset=utf-8
< x-oauth-scopes: pipeline, snippet, issue, pullrequest, project, team, account
< strict-transport-security: max-age=31536000; includeSubDomains; preload
< date: Fri, 12 Jan 2018 15:53:00 GMT
< location: https://bbuseruploads.s3.amazonaws.com/b7f9d00a-c3b1-4451-81d6-157d03db2935/downloads/cc27ac18-7a6e-4a39-b329-23afd4200b65                                                                                                                                         /zip.zip?Signature=ohS6hkNDuO5UcxaWCWJQJUfmeHI%3D&Expires=1515773632&AWSAccessKeyId=AKIAIQWXW6WLXMB5QZAQ&versionId=0eLFTVqqjU                                                                                                                                         xvLXwicZ9PEuLkiL38CPSI&response-content-disposition=attachment%3B%20filename%3D%22zip.zip%22
< x-served-by: app-162
< etag: "d41d8cd98f00b204e9800998ecf8427e"
< x-static-version: e7f50eb4e170
< x-content-type-options: nosniff
< x-accepted-oauth-scopes: repository
< x-credential-type: apppassword
< x-render-time: 0.0570769309998
< x-request-count: 92
< x-frame-options: SAMEORIGIN
< x-version: e7f50eb4e170
< content-length: 0
<
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
* Connection #0 to host api.bitbucket.org left intact
* Issue another request to this URL: 'https://bbuseruploads.s3.amazonaws.com/b7f9d00a-c3b1-4451-81d6-157d03db2935/downloads/cc27ac18-7                                                                                                                                         a6e-4a39-b329-23afd4200b65/zip.zip?Signature=ohS6hkNDuO5UcxaWCWJQJUfmeHI%3D&Expires=1515773632&AWSAccessKeyId=AKIAIQWXW6WLXMB                                                                                                                                         5QZAQ&versionId=0eLFTVqqjUxvLXwicZ9PEuLkiL38CPSI&response-content-disposition=attachment%3B%20filename%3D%22zip.zip%22'
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0*   Trying 52.216.130.19...
* TCP_NODELAY set
* Connected to bbuseruploads.s3.amazonaws.com (52.216.130.19) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [87 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2514 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-SHA
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=Washington; L=Seattle; O=Amazon.com Inc.; CN=*.s3.amazonaws.com
*  start date: Sep 22 00:00:00 2017 GMT
*  expire date: Jan  3 12:00:00 2019 GMT
*  subjectAltName: host "bbuseruploads.s3.amazonaws.com" matched cert's "*.s3.amazonaws.com"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert Baltimore CA-2 G2
*  SSL certificate verify ok.
} [5 bytes data]
> HEAD /b7f9d00a-c3b1-4451-81d6-157d03db2935/downloads/cc27ac18-7a6e-4a39-b329-23afd4200b65/zip.zip?Signature=ohS6hkNDuO5Ucxa                                                                                                                                         WCWJQJUfmeHI%3D&Expires=1515773632&AWSAccessKeyId=AKIAIQWXW6WLXMB5QZAQ&versionId=0eLFTVqqjUxvLXwicZ9PEuLkiL38CPSI&response-content-dis                                                                                                                                         position=attachment%3B%20filename%3D%22zip.zip%22 HTTP/1.1
> Host: bbuseruploads.s3.amazonaws.com
> User-Agent: curl/7.57.0
> Accept: */*
> Testing: Test header so you see this works
>
{ [5 bytes data]
< HTTP/1.1 403 Forbidden
< x-amz-request-id: 63280EEF1A865128
< x-amz-id-2: Ui+My1yzmtT+yjmRjg8eASUW2lUPf1EQUDu08hSXSsYTgPM4RPS9oJ1V0XEen/XodyfJz9aeG4w=
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Fri, 12 Jan 2018 15:53:01 GMT
< Server: AmazonS3
<
  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
* Connection #1 to host bbuseruploads.s3.amazonaws.com left intact

Why are you setting the `content-type` header in your python code, when you're not setting this with `curl`? — larsks, Jan 12 '18 at 15:17
@larsks I can remove that if you like, but the code still gives same error — , Jan 12 '18 at 15:18
You could enable debugging for `requests` as described [here](https://stackoverflow.com/questions/10588644/how-can-i-see-the-entire-http-request-thats-being-sent-by-my-python-application) and compare it to the request that `curl` is making. — larsks, Jan 12 '18 at 15:26
@larsks Great idea. Any idea how I can do this for Curl as well. I am running curl in git bash — , Jan 12 '18 at 15:34
Possibly `curl --trace-ascii=request.log ...` (then examine `request.log` afterwards). — larsks, Jan 12 '18 at 15:48
@larsks That didn't work for me. Do you mean something like: curl -v -I -H — , Jan 12 '18 at 16:00
Drop the `=` (`--trace-ascii request.log`). See the curl man page for details in case I made any other typos. — larsks, Jan 12 '18 at 16:05
I may be looking at this incorrectly, but what is your expected result? Aren't both requests (curl and py) giving out 401 errors? — Nico Villanueva, Jan 12 '18 at 16:09
@larsks Actually, that was right it works now only it downloads the file with and does not record anything. curl -v -I -H is for headers and I appended that in the question — , Jan 12 '18 at 16:10
@NicoVillanueva Right sorry. Curl with gitbash Works perfectly 100% perfecto. Requests does not — , Jan 12 '18 at 16:11
@larsks I've submitted both posts. I'm not sure why this error is coming up. Hmm — , Jan 12 '18 at 16:24

Codyj110 · Answer 1 · 2018-01-12T23:05:03.130

0

You could use the api to download repo from a specific commit like this. If you go to the downloads section in your repo you should see a link similar to this that you can use.

response = requests.get('https://bitbucket.org/{TeamName}/{RepoName/get/{sha}.zip', auth=('User', 'Pass'))

edited Jan 12 '18 at 23:05

answered Jan 12 '18 at 22:03

Codyj110

548
1
5
13

score 0 · Answer 2 · 2018-01-13T02:11:55.247

This worked for me:

import requests

url = 'https://bitbucket.org/UserName/repository/downloads/example.zip'
username = 'user@host.tld'
password = 'secret'
result = requests.get(url, stream=True, auth=(username, password))
if not result.status_code == requests.codes.ok:
    print "ERROR: Could not get %s" % url
else:
    with open('example.zip', 'wb') as zip_file:
        for block in result.iter_content(1024):
            zip_file.write(block)

Notice that I didn't have to set the headers or disable verification.

Curl works, requests doesn't

2 Answers2