8

Here is my script:

import requests, os

ips = ['158.46.169.208','158.46.169.252','158.46.169.76','158.46.171.23','158.46.172.217','158.46.172.55','158.46.172.98','158.46.173.104']
headers =  {"User-Agent": "Edg/90.0.818.56"}

os.system("python3 --version") #On Windows it changes to os.system("python --version")

for i in ips:
    pr = {'http':"http://"+"abcd-"+i+':xyz@example.io:22225','https':'https://'+"abcd-"+i+':xyz@example.io:22225'}
    res1 = requests.get("https://www.google.com/search?q=butter",headers=headers, proxies= pr)
    print(requests.get("https://www.httpbin.org/ip",proxies = pr,headers=headers).text)
    print(res1.status_code)

Output on Windows 10:

Python 3.8.2
{
  "origin": "158.46.169.208"
}

200
{
  "origin": "158.46.169.252"
}

429
{
  "origin": "158.46.169.76"
}

200
{
  "origin": "158.46.171.23"
}

200
{
  "origin": "158.46.172.217"
}

200
{
  "origin": "158.46.172.55"
}

200
{
  "origin": "158.46.172.98"
}

Output on Ubuntu 18.04:

Python 3.8.0
{
  "origin": "158.46.169.208"
}

429
{
  "origin": "158.46.169.252"
}

429
{
  "origin": "158.46.169.76"
}

429
{
  "origin": "158.46.171.23"
}

429
{
  "origin": "158.46.172.217"
}

429
{
  "origin": "158.46.172.55"
}

429
{
  "origin": "158.46.172.98"
}

429
{
  "origin": "158.46.173.104"
}

429

No matter how many times I run the script (even at the same time on two machines), the outputs remain always the same.

I am unable to understand why it blocks the requests on the Ubuntu. And at the same time, it allows the same requests from windows machine while using the same proxy.

I know 429 error means too many requests but then why it gets successful all the time I run on the Windows machine?

EDIT: I have a dual boot laptop, so I logged in to Ubuntu 18.04 on my laptop on which I have windows too. And it seems to work fine here. Same behavior as of Windows. But still it fails when I run it on my server having Ubuntu 18.04 and again everything is same as above.

Mahi
  • 121
  • 1
  • 4
  • Does it have to do with `headers = {"User-Agent": "Edg/90.0.818.56"}`? Does `Edg` mean "Edge" which is a Windows browser and the server doesn't like it when this comes from a Linux machine? – mkrieger1 May 18 '21 at 08:42
  • @mkrieger1 Tried changing it to `Firefox/88.0` but no effect. – Mahi May 18 '21 at 09:42
  • Have you checked this question ? https://stackoverflow.com/questions/22786068/how-to-avoid-http-error-429-too-many-requests-python. – Skaddd May 18 '21 at 10:16
  • Does this answer your question? [How to avoid HTTP error 429 (Too Many Requests) python](https://stackoverflow.com/questions/22786068/how-to-avoid-http-error-429-too-many-requests-python) – Dima Tisnek Jun 09 '21 at 04:41

2 Answers2

0

You need to understand the meaning of HTTP response 200 (HTTP OK) and 429 (Too many requests), as explained in this Wikipedia article.

As you can see, the problem seems not to be due to your program, but to the machine you're working with, who seems to be launching too many requests (at least that's how the server understands it).

In short, you are dealing with a network related issue, you might consider talking about this issue with your network administrator.

Dominique
  • 16,450
  • 15
  • 56
  • 112
  • But they are launching the same number of requests in both cases? – mkrieger1 May 18 '21 at 11:00
  • @mkrieger1 Updated the question please have a look. – Mahi May 18 '21 at 12:27
  • 1
    do your first run it on windows and then via ubuntu? try other way around... as it may be that each of the ips have some rate limit in place - possibly allowing only one request for minute (but that is just a wild guess) – scagbackbone May 25 '21 at 18:12
0

You need to include Accept-Encoding to your headers.

import requests, os
ips = ['158.46.169.208','158.46.169.252','158.46.169.76','158.46.171.23','158.46.172.217','158.46.172.55','158.46.172.98','158.46.173.104']
headers =  {"User-Agent": "Edg/90.0.818.56", "Accept-Handling": "gzip, deflate, br"}

os.system("python3 --version") #On Windows it changes to os.system("python --version")

for i in ips:
    pr = {'http':"http://"+"abcd-"+i+':xyz@example.io:22225','https':'https://'+"abcd-"+i+':xyz@example.io:22225'}
    res1 = requests.get("https://www.google.com/search?q=butter",headers=headers, proxies= pr)
    print(requests.get("https://www.httpbin.org/ip",proxies = pr,headers=headers).text)
    print(res1.status_code)
haritz
  • 1
  • 1