Library `requests` getting different results unpredictably

Question

Why does this code:

import requests


response = requests.post('http://evds.tcmb.gov.tr/cgi-bin/famecgi', data={
    'cgi': '$ozetweb',
    'ARAVERIGRUP': 'bie_yymkpyuk.db',
    'DIL': 'UK',
    'ONDALIK': '5',
    'wfmultiple_selection': 'ZAMANSERILERI',
    'f_begdt': '07-01-2005',
    'f_enddt': '07-10-2016',
    'ZAMANSERILERI': ['TP.PYUK1', 'TP.PYUK2', 'TP.PYUK21', 'TP.PYUK22', 'TP.PYUK3', 'TP.PYUK4', 'TP.PYUK5', 'TP.PYUK6'],
    'YON': '3',
    'SUBMITDEG': 'Report',
    'GRTYPE': '1',
    'EPOSTA': 'xxx',
    'RESIMPOSTA': '***',
})

print(response.text)

produces different results in Python 2 (2.7.12) and Python 3 (3.5.2)? I'm using requests==2.11.1. Since the requests library supports both Python versions with the same API, I guess the result should be the same.

The expected result is the one obtained from running the code with Python 2. It works every single time. When ran with Python 3, the server sometimes returns an error, and sometimes it works. (This is the intriguing part.)

Since it works with Python 2, I figure the error must happen in the client side. Is there any caveat to how Python 3 handles encoding, or sending the data through the socket, that I should be aware of?

EDIT: In the comments below, a person was able to reproduce this and confirms this issue exists.

`POST` usually modifies data on server. Why should you expect it to return the same everytime? — wim, Oct 18 '16 at 21:21
In this case, it doesn't, it's just the submitted data of a public form. Also, with Python 2, it returns the same, every single time. There *is* some inconsistency between Python 2 and 3 behavior here, because the server doesn't know which Python I'm using, still the response changes when I change Python version. — jpmelos, Oct 18 '16 at 21:22
Could you try doing `from __future__ import unicode_literals` and then check the python2 behaviour again. Because without that, it's not actually the same request (python2 has bytes data and python3 has str data) — wim, Oct 18 '16 at 21:25
Either way, the response code is 200, but the string in the body shows an error when using Python 3. — jpmelos, Oct 18 '16 at 21:26
@wim: Even with `from __future__ import unicode_literals`, still works with Python 2 every time. — jpmelos, Oct 18 '16 at 21:28
I've checked the request headers and body, and it apparently is the same with both versions. I am wondering if this could be something in a lower level of the API. — jpmelos, Oct 18 '16 at 21:30
I can reproduce, works fine with python2, does not actually work at all for me using python3. — Padraic Cunningham, Oct 18 '16 at 21:30
Interesting, if you pass `data = what you get if you print response.request.body using python2`, it works and also if you make a list of tuples in place of a dict it works every time. — Padraic Cunningham, Oct 18 '16 at 21:42
Great! I'll make this work using a list of tuples then. I also reported this issue in their GitHub, so I hope they'll see this and check if it's a bug. — jpmelos, Oct 18 '16 at 21:46
The body of the requests are identical as far as the encoding etc.. goes, unless you some reason the order is affecting the output I cannot see what else is different. — Padraic Cunningham, Oct 18 '16 at 21:50
I think we may have a clue, run both of these http://pastebin.com/BRFfEKdE, add a `print(good.text)` and a `print(bad.text)` at the bottom and I bet only the good will output what you want. I think the order does indeed matter, because of how the hashing works in python3 vs python2 is the reason you see the difference — Padraic Cunningham, Oct 18 '16 at 21:54
@PadraicCunningham in English please? Are you saying the server cares about what order the data comes in? — wim, Oct 18 '16 at 21:57
@wim, yes, that is what seems to be the difference, in particular `('cgi', '$ozetweb')` not being first seems to always reproduce the problem. — Padraic Cunningham, Oct 18 '16 at 22:00
I didn't even think of that, since servers are not supposed to care about that order. I guess it's solved now. Padraic, can you write an answer so I can mark it as solved and you get the points? — jpmelos, Oct 18 '16 at 22:02
Yes, this should be in answer not comments :P Somewhat disappointed in the answer.. thought it might have been some crazy requests bug! — wim, Oct 18 '16 at 22:08
@jpmelos, sure, will throw something together in a few minutes. — Padraic Cunningham, Oct 18 '16 at 22:09

score 1 · Accepted Answer · edited May 23 '17 at 12:33

It does seem to come down to different between dicts in python2 vs python3 in relation to Hash randomization is enabled by default since python3.3 and the server needing at least the cgi field to come first, the following can reproduce:

good = requests.post('http://evds.tcmb.gov.tr/cgi-bin/famecgi', data=([
    ('cgi', '$ozetweb'),
    ('ARAVERIGRUP', 'bie_yymkpyuk.db'),
    ('DIL', 'UK'),
    ('ONDALIK', '5'),
    ('wfmultiple_selection', 'ZAMANSERILERI'),
    ('f_begdt', '07-01-2005'),
    ('f_enddt', '07-10-2016'),
    ('ZAMANSERILERI',
     ['TP.PYUK1', 'TP.PYUK2', 'TP.PYUK21', 'TP.PYUK22', 'TP.PYUK3', 'TP.PYUK4', 'TP.PYUK5', 'TP.PYUK6']),
    ('YON', '3'),
    ('SUBMITDEG', 'Report'),
    ('GRTYPE', '1'),
    ('EPOSTA', 'xxx'),
    ('RESIMPOSTA', '***')]))


bad = requests.post('http://evds.tcmb.gov.tr/cgi-bin/famecgi', data=([
    ('ARAVERIGRUP', 'bie_yymkpyuk.db'),
    ('cgi', '$ozetweb'),
    ('DIL', 'UK'),
    ('wfmultiple_selection', 'ZAMANSERILERI'),
    ('ONDALIK', '5'),
    ('f_begdt', '07-01-2005'),
    ('f_enddt', '07-10-2016'),
    ('ZAMANSERILERI',
     ['TP.PYUK1', 'TP.PYUK2', 'TP.PYUK21', 'TP.PYUK22', 'TP.PYUK3', 'TP.PYUK4', 'TP.PYUK5', 'TP.PYUK6']),
    ('YON', '3'),
    ('SUBMITDEG', 'Report'),
    ('GRTYPE', '1'),
    ('EPOSTA', 'xxx'),
    ('RESIMPOSTA', '***')]))

Running the code above using python2:

In [6]: print(good.request.body)
   ...: print(bad.request.body)
   ...: 
   ...: print(len(good.text), len(bad.text))
   ...: 
cgi=%24ozetweb&ARAVERIGRUP=bie_yymkpyuk.db&DIL=UK&ONDALIK=5&wfmultiple_selection=ZAMANSERILERI&f_begdt=07-01-2005&f_enddt=07-10-2016&ZAMANSERILERI=TP.PYUK1&ZAMANSERILERI=TP.PYUK2&ZAMANSERILERI=TP.PYUK21&ZAMANSERILERI=TP.PYUK22&ZAMANSERILERI=TP.PYUK3&ZAMANSERILERI=TP.PYUK4&ZAMANSERILERI=TP.PYUK5&ZAMANSERILERI=TP.PYUK6&YON=3&SUBMITDEG=Report&GRTYPE=1&EPOSTA=xxx&RESIMPOSTA=%2A%2A%2A
ARAVERIGRUP=bie_yymkpyuk.db&cgi=%24ozetweb&DIL=UK&wfmultiple_selection=ZAMANSERILERI&ONDALIK=5&f_begdt=07-01-2005&f_enddt=07-10-2016&ZAMANSERILERI=TP.PYUK1&ZAMANSERILERI=TP.PYUK2&ZAMANSERILERI=TP.PYUK21&ZAMANSERILERI=TP.PYUK22&ZAMANSERILERI=TP.PYUK3&ZAMANSERILERI=TP.PYUK4&ZAMANSERILERI=TP.PYUK5&ZAMANSERILERI=TP.PYUK6&YON=3&SUBMITDEG=Report&GRTYPE=1&EPOSTA=xxx&RESIMPOSTA=%2A%2A%2A
(71299, 134)

And python3:

In [4]: print(good.request.body)
   ...: print(bad.request.body)
   ...: 
   ...: print(len(good.text), len(bad.text))
   ...: 
cgi=%24ozetweb&ARAVERIGRUP=bie_yymkpyuk.db&DIL=UK&ONDALIK=5&wfmultiple_selection=ZAMANSERILERI&f_begdt=07-01-2005&f_enddt=07-10-2016&ZAMANSERILERI=TP.PYUK1&ZAMANSERILERI=TP.PYUK2&ZAMANSERILERI=TP.PYUK21&ZAMANSERILERI=TP.PYUK22&ZAMANSERILERI=TP.PYUK3&ZAMANSERILERI=TP.PYUK4&ZAMANSERILERI=TP.PYUK5&ZAMANSERILERI=TP.PYUK6&YON=3&SUBMITDEG=Report&GRTYPE=1&EPOSTA=xxx&RESIMPOSTA=%2A%2A%2A
ARAVERIGRUP=bie_yymkpyuk.db&cgi=%24ozetweb&DIL=UK&wfmultiple_selection=ZAMANSERILERI&ONDALIK=5&f_begdt=07-01-2005&f_enddt=07-10-2016&ZAMANSERILERI=TP.PYUK1&ZAMANSERILERI=TP.PYUK2&ZAMANSERILERI=TP.PYUK21&ZAMANSERILERI=TP.PYUK22&ZAMANSERILERI=TP.PYUK3&ZAMANSERILERI=TP.PYUK4&ZAMANSERILERI=TP.PYUK5&ZAMANSERILERI=TP.PYUK6&YON=3&SUBMITDEG=Report&GRTYPE=1&EPOSTA=xxx&RESIMPOSTA=%2A%2A%2A
71299 134

Passing your dict as posted in python2:

In [4]: response.request.body
Out[4]: 'cgi=%24ozetweb&DIL=UK&f_enddt=07-10-2016&YON=3&RESIMPOSTA=%2A%2A%2A&wfmultiple_selection=ZAMANSERILERI&ARAVERIGRUP=bie_yymkpyuk.db&GRTYPE=1&SUBMITDEG=Report&f_begdt=07-01-2005&ZAMANSERILERI=TP.PYUK1&ZAMANSERILERI=TP.PYUK2&ZAMANSERILERI=TP.PYUK21&ZAMANSERILERI=TP.PYUK22&ZAMANSERILERI=TP.PYUK3&ZAMANSERILERI=TP.PYUK4&ZAMANSERILERI=TP.PYUK5&ZAMANSERILERI=TP.PYUK6&ONDALIK=5&EPOSTA=xxx'

In [5]: len(response.text)
Out[5]: 71299

And the same dict in python3:

In [3]: response.request.body
Out[3]: 'EPOSTA=xxx&ARAVERIGRUP=bie_yymkpyuk.db&DIL=UK&SUBMITDEG=Report&cgi=%24ozetweb&GRTYPE=1&f_enddt=07-10-2016&wfmultiple_selection=ZAMANSERILERI&ONDALIK=5&f_begdt=07-01-2005&RESIMPOSTA=%2A%2A%2A&YON=3&ZAMANSERILERI=TP.PYUK1&ZAMANSERILERI=TP.PYUK2&ZAMANSERILERI=TP.PYUK21&ZAMANSERILERI=TP.PYUK22&ZAMANSERILERI=TP.PYUK3&ZAMANSERILERI=TP.PYUK4&ZAMANSERILERI=TP.PYUK5&ZAMANSERILERI=TP.PYUK6'

In [4]: len(response.text)
Out[4]: 134

And running ~$ export PYTHONHASHSEED=1234 before starting another ipython2 shell:

In [4]: response.request.body
Out[4]: 'DIL=UK&GRTYPE=1&ARAVERIGRUP=bie_yymkpyuk.db&f_begdt=07-01-2005&RESIMPOSTA=%2A%2A%2A&ONDALIK=5&EPOSTA=xxx&YON=3&SUBMITDEG=Report&wfmultiple_selection=ZAMANSERILERI&cgi=%24ozetweb&ZAMANSERILERI=TP.PYUK1&ZAMANSERILERI=TP.PYUK2&ZAMANSERILERI=TP.PYUK21&ZAMANSERILERI=TP.PYUK22&ZAMANSERILERI=TP.PYUK3&ZAMANSERILERI=TP.PYUK4&ZAMANSERILERI=TP.PYUK5&ZAMANSERILERI=TP.PYUK6&f_enddt=07-10-2016'

In [5]: os.environ["PYTHONHASHSEED"]
Out[5]: '1234'
In [6]: len(response.text)
Out[6]: 134

You can run the code numerous times to the same end but definitely ('cgi', '$ozetweb') coming first is essential for the code to work, it happened to work using python3 intermittently as the order of the keys sometimes put cgi first. There is a bit more on the hashing topic

Library `requests` getting different results unpredictably

1 Answers1