6

I ran into a strange behavior of the requests package ... or so I believe:

(1) for a certain website, requests.get runs on Windows, but it freezes on Linux (or raises timeout error if timeout is set to several seconds)

(2) urllib.request.urlopen works on both on Windows and Linux

Does any body have a suggestions how to debug/investigate this further?

Here are the details:

On my Windows machine I run Inside a Jupyter Notebook:

import sys
import requests
print(sys.version)
print(requests.__version__)

3.6.3 |Anaconda custom (32-bit)| (default, Oct 15 2017, 07:29:16) [MSC v.1900 32 bit (Intel)]

2.18.4

Then I try:

time1 = time.time()
response=requests.get('https://www.tagheuer.com/en/watches/tag-heuer-carrera-calibre-5-automatic-watch-39-mm-war211a-ba0782')
time2 = time.time()
print(response.status_code)
print(time2-time1)

with the result:

200

0.6808576583862305

So far, so well ... then I move to the Unix server:

import sys
import requests
print(sys.version)
print(requests.__version__)

3.5.2 (default, Nov 23 2017, 16:37:01) \n[GCC 5.4.0 20160609]

2.18.4

when I issue the request statement,

response=requests.get('https://www.tagheuer.com/en/watches/tag-heuer-carrera-calibre-5-automatic-watch-39-mm-war211a-ba0782')

it takes ages to come back. Sometimes returning bytes that does not look like the requested page at all. Sometimes I run out of patience and hit CTRL-C.

But on the same server:

 from urllib.request import urlopen
 import time
 url='https://www.tagheuer.com/en/watches/tag-heuer-carrera-calibre-5-automatic-watch-39-mm-war211a-ba0782'
 time1 = time.time()
 f = urlopen(url)
 myfile = f.read()
 time2 = time.time()
 print(f.status)
 print(time2-time1)

finishes with the result

200

0.22718000411987305

Why would the requests library let me down on the Linux machine but not the Windows box?

Why would urllib.request.urlopen work but requests would freeze? What are they doing differently?

The same code works for www.google.com on both platforms. Which (non-standard?) behavior of the TAG_Heuer website causes requests to fail?

Looking forward to your feedback and ideas!

Olaf
  • 145
  • 1
  • 1
  • 12
  • 1
    [An answer on a similar question](https://stackoverflow.com/a/43987762/877069) suggests that it may be related to the site using user-agent detection. If you add a user-agent header, does it change anything? – Nick Chammas Apr 15 '18 at 23:30
  • 1
    Here's [another answer](https://stackoverflow.com/a/45768596/877069) suggesting something similar. – Nick Chammas Apr 15 '18 at 23:31

0 Answers0