Making a POST request using urllib with multiple headers gives 400 Bad Request error

Question

I have used requests library and I know how to work with it, but I need to work with standard library only, so I would appreciate if you don't encourage me to use requests instead.

I made a flask server that handles POST requests and then from a different script I call urllib to make POST calls to the flask server. I need to send a raw json in body just like we do in Postman.

Flask Server

from flask import Flask, request

app = Flask(__name__)
@app.route('/random', methods=['POST'])
def random():
    if request.method == 'POST':
        if request.headers.get('Authorization') and request.headers.get('Content-Type') == 'application/json':
            print(request.get_json())
            return "Success"
        else:
            print(request.get_json())
            return "Bad request"

app.run(host='0.0.0.0', port=5000, debug=True)

Urllib Client (saved as test.py) -

import urllib.request
import urllib.parse
d = {"spam": 1, "eggs": 2, "bacon": 0}
data = urllib.parse.urlencode(d)
data = data.encode()
req = urllib.request.Request("http://localhost:5000/random", data)
req.add_header('Content-Type', 'application/json')
req.add_header('Authorization', 12345)
with urllib.request.urlopen(req) as f:
    print(f.read().decode('utf-8'))

With only Authorization header I get Bad Request as output and the json is None on the flask server side as expected.

With ONLY Content-Type header OR both the headers I get this error -

Traceback (most recent call last):
  File "test.py", line 9, in <module>
    with urllib.request.urlopen(req) as f:
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 532, in open
    response = meth(req, response)
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 570, in error
    return self._call_chain(*args)
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 504, in _call_chain
    result = func(*args)
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: BAD REQUEST

The whole thing is simple enough, but I am not able to understand why is this happening and the error message doesn't help much either.

score 3 · Accepted Answer · answered Nov 27 '18 at 13:21

3

The server is failing in request.get_json(). It's only happening when the client sends both headers because that's when it reaches this line.

To fix it, change the client to send the data as JSON:

import json            # <-- Import json
import urllib.request
import urllib.parse

d = {"spam": 1, "eggs": 2, "bacon": 0}
data = json.dumps(d)   # <-- Dump the dictionary as JSON
data = data.encode()
req = urllib.request.Request("http://localhost:5000/random", data)
req.add_header('Content-Type', 'application/json')
req.add_header('Authorization', 12345)
with urllib.request.urlopen(req) as f:
    print(f.read().decode('utf-8'))

I hope this helps

answered Nov 27 '18 at 13:21

Carlos Mermingas

3,822
2
21
40

This does solve the issue, however I am not able to understand why do i need to use `json.dumps` as the [docs](https://docs.python.org/3/library/urllib.request.html) (third from last example before **Legacy interface**) and a few other SO posts suggest to use `urlencode` like i have ```d = {"spam": 1, "eggs": 2, "bacon": 0} data = urllib.parse.urlencode(d) data = data.encode()``` – jar Nov 27 '18 at 13:30
The SO posts I am talking about are, [this](https://stackoverflow.com/a/41917763/3647970) and [this](https://stackoverflow.com/questions/36484184/python-make-a-post-request-using-python-3-urllib) – jar Nov 27 '18 at 13:34
The first link in your comment did not solve the problem. The [accepted answer](https://stackoverflow.com/a/41918479/6118299) uses the `requests` library, which would work for you, too and I recommend it. The second link in your comment works with `urlencode` because it's simulating POSTing a form. The data in a POST request is just a big string. The encoding tells the server how to interpret that data. Try printing the raw request (`print request.data`) instead of using `get_json()` to see the difference between URL-encoded data and JSON. – Carlos Mermingas Nov 28 '18 at 12:19
I finally understand it now. Thanks a lot for the time and effort. However I was playing around with what you suggested and I found this peculiar thing, if i don't pass any of the headers from the urllib script but i do pass the json then the server receives an empty byte like `b''` . However when I do the same via POSTMAN, i.e manually disable the headers by un-ticking them but keep the json body as it is, I still somehow get the whole json byte as `b'{"spam": 1, "eggs": 2, "bacon": 0}'`. Any idea why is this happening? – jar Nov 28 '18 at 13:32

Making a POST request using urllib with multiple headers gives 400 Bad Request error

1 Answers1

Linked