2

Python requests not working for a website API but works in Browser (chrome). it used to work earlier but from the past 2 months its not working. the api i am using is yts.mx api.

I'll attach the code snippet below:

import requests
a = requests.get('https://yts.mx/api/v2/list_movies.json')

This is the error i am getting:

Traceback (most recent call last):
  File "D:\Programs\Python\Python39\lib\site-packages\urllib3\connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "D:\Programs\Python\Python39\lib\site-packages\urllib3\connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "D:\Programs\Python\Python39\lib\site-packages\urllib3\connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "D:\Programs\Python\Python39\lib\http\client.py", line 1347, in getresponse
    response.begin()
  File "D:\Programs\Python\Python39\lib\http\client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "D:\Programs\Python\Python39\lib\http\client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "D:\Programs\Python\Python39\lib\socket.py", line 704, in readinto
    return self._sock.recv_into(b)
  File "D:\Programs\Python\Python39\lib\ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "D:\Programs\Python\Python39\lib\ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Programs\Python\Python39\lib\site-packages\requests\adapters.py", line 439, in send
    resp = conn.urlopen(
  File "D:\Programs\Python\Python39\lib\site-packages\urllib3\connectionpool.py", line 726, in urlopen
    retries = retries.increment(
  File "D:\Programs\Python\Python39\lib\site-packages\urllib3\util\retry.py", line 410, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "D:\Programs\Python\Python39\lib\site-packages\urllib3\packages\six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "D:\Programs\Python\Python39\lib\site-packages\urllib3\connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "D:\Programs\Python\Python39\lib\site-packages\urllib3\connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "D:\Programs\Python\Python39\lib\site-packages\urllib3\connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "D:\Programs\Python\Python39\lib\http\client.py", line 1347, in getresponse
    response.begin()
  File "D:\Programs\Python\Python39\lib\http\client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "D:\Programs\Python\Python39\lib\http\client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "D:\Programs\Python\Python39\lib\socket.py", line 704, in readinto
    return self._sock.recv_into(b)
  File "D:\Programs\Python\Python39\lib\ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "D:\Programs\Python\Python39\lib\ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
urllib3.exceptions.ProtocolError: ('Connection aborted.', TimeoutError(10060, 'A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond', None, 10060, None))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Programs\Python\Python39\test.py", line 8, in <module>
    a = r.get('https://yts.mx/api/v2/list_movies.json')
  File "D:\Programs\Python\Python39\lib\site-packages\requests\api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "D:\Programs\Python\Python39\lib\site-packages\requests\api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "D:\Programs\Python\Python39\lib\site-packages\requests\sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "D:\Programs\Python\Python39\lib\site-packages\requests\sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "D:\Programs\Python\Python39\lib\site-packages\requests\adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', TimeoutError(10060, 'A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond', None, 10060, None))

and sometimes getting error like forcibly closed due to existing connection, this is the trace:

Traceback (most recent call last):
  File "D:\Programs\Python\Python39\lib\site-packages\urllib3\connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "D:\Programs\Python\Python39\lib\site-packages\urllib3\connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "D:\Programs\Python\Python39\lib\site-packages\urllib3\connectionpool.py", line 978, in _validate_conn
    conn.connect()
  File "D:\Programs\Python\Python39\lib\site-packages\urllib3\connection.py", line 362, in connect
    self.sock = ssl_wrap_socket(
  File "D:\Programs\Python\Python39\lib\site-packages\urllib3\util\ssl_.py", line 386, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "D:\Programs\Python\Python39\lib\ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "D:\Programs\Python\Python39\lib\ssl.py", line 1040, in _create
    self.do_handshake()
  File "D:\Programs\Python\Python39\lib\ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

This used to perfectly work a month ago or so. Tried various methods seen on web.

UPDATE

Turns out the issue is with the IP . When i use a VPN connection it works perfectly. However i dont need a VPN to access it from my web browser. any Idea why this is happening? is there a way to bypass this?

django
  • 41
  • 1
  • 5
  • Works as expected for me. Looks like the problem is in the connection, not in the code. – Thierry Lathuille Dec 21 '20 at 08:30
  • i've tried from multiple systems different locations. still no luck – django Dec 21 '20 at 08:31
  • Are you behind a proxy server by any chance? – Vikash Balasubramanian Dec 21 '20 at 08:56
  • @VikashB no I'm not. – django Dec 21 '20 at 09:02
  • Have you tried it after Thierry Lathuille posted his comment? It could be, that the sever vas temporarily down, and you just had a bad timing. I tried it like 2 minutes ago and everything works as expected. Is it also possible, that your python executable is blocked by your firewall, since you it works with your browser, but not with your python module. What OS are you using? – captainmoron Dec 21 '20 at 11:54
  • @captainmoron hey, turns out the problem was with my ip. when i use VPN , the request is successful. however i dont need a VPN connecion to access it from browser. any Idea? – django Dec 21 '20 at 12:17
  • This is more of a reach, but where are you from? Maybe your contry or ISP blocked the web site. It is a torrent page and that happens sometimes. – captainmoron Dec 21 '20 at 12:29
  • @captainmoron I'm from India and yes that's the case. but i can access it from my browser. i checked the web for a workaround and most of them are proxies. is there any other work around :/ – django Dec 21 '20 at 12:32

1 Answers1

4

Since you mentioned, that the browser request goes through but the python request does not, it crossed my mind, that the web site iteself blocks non-browser-requests. I found this question here, that dealt with that problem. The solution was to add a custom header to the python request to make it look like a browser request:

import requests
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.76 Safari/537.36'} # This is chrome, you can set whatever browser you like
url = 'https://yts.mx/api/v2/list_movies.json'
a = requests.get(url,headers)
print(a.content)

With your request, you then send the cutom header as the second parameter in your get-request. I tried the script above and it worked for me, but also your original script worked on my connection.

captainmoron
  • 270
  • 2
  • 6
  • same case :/. as i said the issue is only resolved when i connect to a VPN or pass a proxy. but proxies are not permanent. it's hard to change the source code each time the proxy is outcasted. – django Dec 21 '20 at 16:57
  • I am currently researching, if this could be solved with proxies embedded in python. Maybe this could help ( https://www.scrapehero.com/how-to-rotate-proxies-and-ip-addresses-using-python-3/ ) Since it is Christmas time, I will try to formulate something in a few days. – captainmoron Dec 24 '20 at 14:32