2

I am using the Python Requests module (v. 2.19.1) with Python 3.4.3, calling a function on a remote server that generates a .csv file for download. In general, it works perfectly. There is one particular file that takes >6 minutes to complete, and no matter what I set the timeout parameter to, I get an error after exactly 5 minutes trying to generate that file.

import requests
s = requests.Session()
authPayload = {'UserName': 'myloginname','Password': 'password'}
loginURL = 'https://myremoteserver.com/login/authenticate'
login = s.post(loginURL, data=authPayload)
backupURL = 'https://myremoteserver.com/directory/jsp/Backup.jsp'
payload = {'command': fileCommand}
headers = {'Connection': 'keep-alive'}
post = s.post(backupURL, data=payload, headers=headers, timeout=None)

This times out after exactly 5 minutes with the error:

File "/usr/lib/python3/dist-packages/requests/adapters.py", line 330, in send timeout=timeout

File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 612, in urlopen raise MaxRetryError(self, url, e)

urllib3.exceptions.MaxRetryError: > HTTPSConnectionPool(host='myremoteserver.com', port=443): Max retries exceeded with url: /directory/jsp/Backup.jsp (Caused by < class 'http.client.BadStatusLine'>: '')

If I set timeout to something much smaller, say, 5 seconds, I get a error that makes perfect sense:

urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='myremoteserver.com', port=443): Read timed out. (read timeout=5)

If I run the process from a browser, it works fine, so it doesn't seem like it's the remote server closing the connection, or a firewall or something in-between closing the connection.

itnAAnti
  • 652
  • 8
  • 17
  • Have you tried faking a browser by adding a User-Agent header? See the following [SO question and answer](https://stackoverflow.com/questions/27652543/how-to-use-python-requests-to-fake-a-browser-visit) for an example – tatlar Sep 10 '18 at 02:28
  • @tatlar I had not previously tried that, so I tried it now. Unfortunately, it did not affect the issue at all. – itnAAnti Sep 10 '18 at 04:22
  • This link: https://github.com/requests/requests/issues/2364 leads me to believe it /is/ the connection getting cut off by a firewall somewhere (contrary to my comment in the question). I tried adding the Content-Type header that helped in that post, but it did not affect my problem. – itnAAnti Sep 10 '18 at 05:55
  • It looks like your problems are bubbling up to `requests` from the underlying `urllib3` library. You might get more fruitful troubleshooting results by searching for `urllib3` timeout problems rather than `requests`. – tatlar Sep 10 '18 at 06:22
  • This also might be the problem: `http.client.BadStatusLine`. Have you seen this [SO answer](https://stackoverflow.com/questions/18478013/python-requests-exceptions-connectionerror-max-retries-exceeded-with-url)? – tatlar Sep 10 '18 at 16:37
  • 1
    I think that link is the problem/solution. I just tried the script from my laptop with a cellular data connection, and it succeeded, so I think my hardware router/firewall is killing the connection after 5 minutes. – itnAAnti Sep 10 '18 at 17:04
  • 1
    Ugh, that was exactly right. I didn't realize that my router/firewall had an https proxy built-in. I disabled the HTTPS proxy, and everything is working. @tatlar will you please add your recent SO link as an answer, and I'll accept it. Thanks for your help! – itnAAnti Sep 10 '18 at 17:23

1 Answers1

1

Posted at the request of the OP -- my comments on the original question pointed to a related SO problem

The clue to the problem lies in the http.client.BadStatusLine error.

Take a look at the following related SO Q & A that discusses the impact of proxy servers on HTTP requests and responses.

tatlar
  • 3,080
  • 2
  • 29
  • 40