bad request error 400

Question

I want to replicate the below with Client config.

Curl -D- -X GET -H "Authorization: Basic ZnJlZDpmcmVk" -H "Content-Type: application/json" http://localhost:7990/rest/api/1.0/projects
https://developer.atlassian.com/server/bitbucket/how-tos/example-basic-authentication/

This is to download a file for updating exe in a private repo.

I'm getting 400 errors on a regular basis with bitbucket.

What I had:

client = Client(ClientConfig(), headers={'basic_auth':'U:P' }, refresh=True)

I want to include this header

header = {'Content-Type': 'application/json'}

So something like:

client = Client(ClientConfig(), headers={'basic_auth': 'brofewfefwefewef:EKAXsWkdt5H6yJEmtexN'}, header = {'Content-Type': 'application/json'}, refresh=True)

Should fix?

At least according to....

"Some http client software expects to receive an authentication challenge before it will send an authorization header and this may mean that it may not behave as expected. In this case you may need to configure it to supply the authorization header as described above rather than relying on its default mechanism."

and...

https://stackoverflow.com/questions/8840303/urllib2-http-error-400-bad-request

For me though, I always get error even with this change 400 https://pastebin.com/V9ibxTRX (full code here or short version below)

Optional errors message (it's not pretty but I did reduce it):

~~~~~ACCESSING PAGE AND FOUND!~~~
DEBUG:pyupdater.client.downloader:Url for request: https://api.bitbucket.org/2.0/repositories/Username/repository/downloads/keys.gz
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.bitbucket.org
send: b'GET /2.0/repositories/Username/repository/downloads/keys.gz HTTP/1.1\r\nHost: api.bitbucket.org\r\nAccept-Encoding: identity\r\nauthorization: Basic ywafdwafawffwawffawafwfwaawfawfg==\r\n\r\n'
reply: 'HTTP/1.1 302 Found\r\n'

~~~~~~RETRYING REDIRECTION PRESENT (IS THIS THE CAUSE OF ISSUES)??~~
DEBUG:urllib3.connectionpool:https://api.bitbucket.org:443 "GET /2.0/repositories/Username/repository/downloads/keys.gz HTTP/1.1" 302 0
DEBUG:urllib3.util.retry:Incremented Retry for (url='https://api.bitbucket.org/2.0/repositories/Username/repository/downloads/keys.gz'): Retry(total=2, connect=None, read=None, redirect=None, status=None)
INFO:urllib3.poolmanager:Redirecting https://api.bitbucket.org/2.0/repositories/Username/repository/downloads/keys.gz -> https://bbuseruploads.s3.amazonaws.com/a0e395b6-0c54-4efb-9074-57ec4190020b/downloads/1c87431a-98de-4d97-8c80-000243f81cba/keys.gz?Signature=FvA9X7K9ryM2Ft2mTV7PZefidJY%3D&Expires=1515817377&AWSAccessKeyId=AKIAIQWXW6WLXMB5QZAQ&versionId=6J830UBC1RFvWz.R6pMDwIiJQNKJjSkm&response-content-disposition=attachment%3B%20filename%3D%22keys.gz%22

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): bbuseruploads.s3.amazonaws.com

~~HEADERS MIGHT BE ISSUE ACCORDING TO DOCS  https://developer.atlassian.com/server/bitbucket/how-tos/example-basic-authentication/~~

header: Server header: Vary header: Content-Type header: X-OAuth-Scopes header: Strict-Transport-Security header: Date header: Location header: X-Served-By header: ETag header: X-Static-Version header: X-Content-Type-Options header: X-Accepted-OAuth-Scopes header: X-Credential-Type header: X-Render-Time header: Connection header: X-Request-Count header: X-Frame-Options header: X-Version header: Content-Length send: b'GET /a0e395b6-0c54-4efb-9074-57ec4190020b/downloads/1c87431a-98de-4d97-8c80-000243f81cba/keys.gz?Signature=FvA9X7K9ryM2Ft2mTV7PZefidJY%3D&Expires=1515817377&AWSAccessKeyId=AKIAIQWXW6WLXMB5QZAQ&versionId=6J830UBC1RFvWz.R6pMDwIiJQNKJjSkm&response-content-disposition=attachment%3B%20filename%3D%22keys.gz%22 HTTP/1.1\r\nHost: bbuseruploads.s3.amazonaws.com\r\nAccept-Encoding: identity\r\nauthorization: Basic YnJvZmV3ZmVmd2VmZXdlZjpFS0FYc1drZHQ1SDZ5SkVtdGV4Tg==\r\n\r\n'
DEBUG:urllib3.connectionpool:https://bbuseruploads.s3.amazonaws.com:443 "GET /a0e395b6-0c54-4efb-9074-57ec4190020b/downloads/1c87431a-98de-4d97-8c80-000243f81cba/keys.gz?
Signature=FvA9X7K9ryM2Ft2mTV7PZefidJY%3D&Expires=1515817377&AWSAccessKeyId=AKIAIQWXW6WLXMB5QZAQ&versionId=6J830UBC1RFvWz.R6pMDwIiJQNKJjSkm&response-content-disposition=attachment%3B%20filename%3D%22keys.gz%22 HTTP/1.1" 400 None

~~UNABLE TO ACCESS PAGE (WAIT, BEFORE IT HAD 'HTTP/1.1 302 Found\r\n' SEE TOP)~~
reply: 'HTTP/1.1 400 Bad Request\r\n'

DEBUG:pyupdater.client.downloader:Resource URL: https://api.bitbucket.org/2.0/repositories/Username/repository/downloads/keys.gz
DEBUG:pyupdater.client.downloader:Got content length of: None
DEBUG:pyupdater.client.downloader:Content-Length not in headers
DEBUG:pyupdater.client.downloader:Callbacks will not show time left or percent downloaded.
DEBUG:pyupdater.client.downloader:Using file as storage since the file is too large

I very much doubt that you are using urllib3 - ther's neither a class `Client` nor `ClientConfig` in the urllib3 source code. Can you post [a **complete** example](https://stackoverflow.com/help/mcve)? — phihag, Jan 13 '18 at 15:23
@phihag Error message full: https://pastebin.com/V9ibxTRX ctr find F -> 400 . If you really want to reproduce you can but it will take more than 5 minutes. Working example here. https://stackoverflow.com/questions/48047296/pyinstaller-updater-with-github-bitbucket-private-repos Line 366-368 shows downloader using url lib https://github.com/JMSwag/PyUpdater/blob/master/pyupdater/client/downloader.py Docs that walk you through it. http://www.pyupdater.org/usage-cli/ Once that’s done, you’ll receive a similar error. — , Jan 13 '18 at 15:49
@phihag Rather than do that, take a look at the example I got from their docs. I think replicating that with config should fix. I have no clue sadly and have been stuck for all day but this is likely how to get around — , Jan 13 '18 at 16:00
Sorry, but none of the posts you mention are anywhere close to complete. Also, the formats don't fit together: In this question you initialize Client with `auth` and a list as `headers`. In the [code you link to](https://github.com/JMSwag/PyUpdater/blob/master/pyupdater/client/__init__.py#L64), Client has no `auth` parameter, and `headers` takes a dict, not a list. If you want anybody to be able to answer your question, you must include the complete code; i.e. code that can be pasted into a file and run immediately. Remove all unnecessary cruft. — phihag, Jan 13 '18 at 16:27
@phihag I have update the question based on what you have explained to me. If it is still not right, just say so and I will clarify — , Jan 13 '18 at 17:19
Why don't you add the Content-Type header to your `headers` dict? — t.m.adam, Jan 13 '18 at 18:01
@t.m.adam I would love to but I cn't quite visualise how that would look. Would you be willing to provide an example? Thanks — , Jan 13 '18 at 22:24

t.m.adam · Accepted Answer · 2018-01-14T11:10:48.867

0

I'm not familiar with the library you're using for your client object, but you should be able to set the Content-Type header in your headers dictionary.

headers = {
    'basic_auth': 'brofewfefwefewef:EKAXsWkdt5H6yJEmtexN', 
    'Content-Type': 'application/json'
}
client = Client(ClientConfig(), headers=headers, refresh=True)

Unfortunately this doesn't seem to be possible because headers is passed (unpacked) to urllib3.util.make_headers and it does not accept a content_type argument.

Also you can't access FileDownloader._http.headers in Client, because it's a local variable.
A possible FileDownloader 'hack':

class FileDownloader(object):
    ...Line 152...
    def _get_http_pool(self, secure=True):
        if secure:
            _http = urllib3.PoolManager(cert_reqs=str('CERT_REQUIRED'), 
                                        ca_certs=certifi.where())
        else:
            _http = urllib3.PoolManager()

        if self.headers:
            content_type = self.headers.get('Content-Type') 
            if 'Content-Type' in self.headers:
                del self.headers['Content-Type']
            _headers = urllib3.util.make_headers(**self.headers)
            _http.headers.update(_headers)
            if content_type:
                _http.headers['content-type'] = content_type
        print(_http.headers)
        return _http

This should allow you to pass a Content-Type header in Client.
I don't recommend modifying the source code of your libs, but if you have no other choice...

edited Jan 14 '18 at 11:10

answered Jan 13 '18 at 22:55

t.m.adam

15,106
3
32
52

Bitbucket, is but a cruel mistress to work with. I get: File "C:\Users\Django\AppData\Local\Continuum\miniconda3\lib\site-packages\pyupdater\client\downloader.py", line 160, in _get_http_pool # _headers = urllib3.util.make_headers(**self.headers) #TypeError: make_headers() got an unexpected keyword argument 'Content-Type'. . So obviously I need to find a way to modify https://github.com/JMSwag/PyUpdater/blob/master/pyupdater/client/downloader.py so that it allows this. Huh.. – Jan 13 '18 at 23:11
Let me take a look at the library, and if i find something useful i'll let you know. – t.m.adam Jan 13 '18 at 23:34
I don't think it's possible to have a Content-Type header in `headers`. It uses `urllib3.util.make_headers` to create headers, and it accepts only those parameters: `keep_alive, accept_encoding, user_agent, basic_auth, proxy_basic_auth, disable_cache`. Perhaps you could monkey patch `make_headers` or use some other hack, although i wouldn't recommend it. There has to be a proper way to set the Content-Type header. – t.m.adam Jan 14 '18 at 00:42
Maybe adding the headers somehow in downloader.py will fix it. As basic auth should work but I think requires those content type to work or it will kick you out. I'm assuming anyway – Jan 14 '18 at 04:41
I need this for work aroundthe office, but it seems no one knows. Api's are painful to work with – Jan 14 '18 at 08:38
Perhaps you could use `requests` or `urllib` for this request? – t.m.adam Jan 14 '18 at 08:52
I've read just about everything I can. I don't understand the library enough to implement any solution. If you could find solution I would be greatful. I not sure what to tell bosses, as we are to use bitbucket and not github anymore :/. Not sure. – Jan 14 '18 at 08:58
The library is in urllib3 so not sure if you can use requests. If possible that would work, requests actually makes sense to me. There's also a million and one answers https://www.reddit.com/r/learnpython/comments/7pweds/downloading_a_file_password_protected_private/?utm_content=title&utm_medium=hot&utm_source=reddit&utm_name=learnpython . Uh well. Laughably I might have better rewriting my own downloader.py then understanding it (assuming I have time) – Jan 14 '18 at 09:03
Error 400... That usually means malformed.. doesn't it. Maybe.. hmm – Jan 14 '18 at 09:15
400 with `requests`? It should be that smple: `r = requests.get(url, auth=(usr,pwd), headers={'Content-Type': 'application/json'})`. – t.m.adam Jan 14 '18 at 09:42
Hmmm.. how can I apply that to the pyupdater though script itself or even client config in this question. :S. That works I think ( I have a similar script that downloads). But I guess I don't see what to adjust in the pyupdater downloader.py – Jan 14 '18 at 09:53
I couldn't find the source code for `ClientConfig` so i don't know what parameters it accepts and its properties, can you help me with that? About the `FileDownloader`, i suppose you could 'hack' it. Try updating `FileDownloader._http.headers` manually, after the object is created. – t.m.adam Jan 14 '18 at 10:18
So what am I including sorry? Could you provide more of example of what I am putting in there in your answer? Pyupdatr config - https://github.com/JMSwag/PyUpdater/blob/master/pyupdater/utils/config.py is this what you meant? Hmmm.. wait... that does no explain a lot sorry that just talks about what pyupdater reads in config file – Jan 14 '18 at 10:21
Sorry, I don't really know how I would implement those changes. Could you give me more of a hint or even an example? I appreciate it.. and I know my work mates will ha – Jan 14 '18 at 10:37
I copied that exactly and still I get fobidden message O; – Jan 14 '18 at 11:33
But no exceptions yes? What do you get from `print(_http.headers)` and from the error logs? Is the Content-Type header there, and is it set to 'application/json'? – t.m.adam Jan 14 '18 at 11:40
I don't think they are being read. Here is exactly what I see: http://textuploader.com/dmkjl – Jan 14 '18 at 11:46
I was to set content type: 'content-Type' : 'application/json' in the main.py not in the downloader.py, right? Could you provide updated version to be sure and I'll double check :). adding that to self.headers.get would create unshashable type dict – Jan 14 '18 at 11:53
Uh okay, yes it's not printing that at all – Jan 14 '18 at 12:02
You've replaced `downloader.FileDownloader._get_http_pool` with my version, and passing a `headers` dict with `'basic_auth'` and `'Content-Type'` keys in `Client` (like in my first code snippet), right? This should either create a Content-Type header or throw an exception. I don't know.. – t.m.adam Jan 14 '18 at 12:20
I copy pasted what you put in your answer as well as copy pasted your initial answer headers = { 'basic_auth': 'brofewfefwefewef:EKAXsWkdt5H6yJEmtexN', 'Content-Type': 'application/json' – Jan 14 '18 at 12:21
Take a look. It's like it selectively works - http://textuploader.com/dmklr – Jan 14 '18 at 12:28
Good news is initial request it display json, bad news is it did not fix and stop displaying later. What are other reasons you get error 400. Hmm... – Jan 14 '18 at 12:30
Any ideas? I really miss Github :S – Jan 14 '18 at 12:43
It seems like the first couple of requests have both authorization and content-type headers, but later only authorization. And the 2nd response is 400 even though the headers look fine. I'm scratching my head here too.. – t.m.adam Jan 14 '18 at 13:02
Yeah haha, quite strange I think it's url lib or json related. 99% of error 400 questions seem to be resolved that way at least. Hmmm.... – Jan 14 '18 at 13:10
According to my repo it downloads keys and versions before bugging out – Jan 14 '18 at 13:20
I'm going to have to convince my boss to use github aren't I? o.o – Jan 14 '18 at 13:30
If that's an option.. Sorry but i'm out of ideas at this point. – t.m.adam Jan 14 '18 at 13:44
I've made added a bounty for a seperate related question here. I suspect this will be one of those questions that may never be solved https://stackoverflow.com/questions/48161651/authenticates-then-redirects-then-gives-400-bad-request-for-bitbucket – Jan 15 '18 at 12:24
We were on the right track. It works for bitbucket, but note in the error log it uploads to amazon with simple auth. Amazon does not support simple auth. I've added a 200 bounty here on it https://stackoverflow.com/questions/48218119/redirect-with-no-headers?noredirect=1&lq=1 – Jan 16 '18 at 08:15

bad request error 400

1 Answers1