1

I am using module google search for webscraping, but I got this error 429. I tried uninstall and install module again, but it didn't help. So my next idea is delete cookies, but I don't know how. Can you help me, please?

query = 'site:https://stackoverflow.com urllib.error.HTTPError: HTTP Error 429: Too Many Requests'
search_query = search(query=query, stop=10)
for url in search_query:
print(url)
DisappointedByUnaccountableMod
  • 6,656
  • 4
  • 18
  • 22
Žanet
  • 64
  • 1
  • 1
  • 7
  • Stackoverflow have ratelimited you due to sending too many requests. This is to prevent DoS attacks. The only way around this is use of proxies, but a better idea is to write a program that simply doesn't send requests very quickly – clubby789 Feb 11 '20 at 13:47
  • _I tried uninstall and install module again, but it didn't help._ Why would that help, though? To quote the [Mozilla developer docs](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429): _The HTTP 429 Too Many Requests response status code indicates the user has sent too many requests in a given amount of time ("rate limiting"). A Retry-After header might be included to this response indicating how long to wait before making a new request._ – AMC Feb 11 '20 at 20:19
  • _So my next idea is delete cookies, but I don't know how._ I'm sure there are plenty of existing resources for this. – AMC Feb 11 '20 at 20:20

2 Answers2

0

429 Too Many Requests

The HTTP 429 Too Many Requests response status code indicates that the user has sent too many requests in a given amount of time ("rate limiting"). The response representations SHOULD include details explaining the condition, and MAY include a Retry-After header indicating how long to wait before making a new request.

Note that this specification does not define how the origin server identifies the user, nor how it counts requests. For example, an origin server that is limiting request rates can do so based upon counts of requests on a per-resource basis, across the entire server, or even among a set of servers. Likewise, it might identify the user by its authentication credentials, or a stateful cookie.

You're sending too many requests in a short period of time. The Custom Search API might be useful depending on your use scenario. If not then you might have to use proxies for your calls, or implement a wait and retry mechanism

Community
  • 1
  • 1
Kasem Alsharaa
  • 892
  • 1
  • 6
  • 15
  • 1
    What is the limit of one query? How many query per time? I got this error for one query, than I changed the content of query, used google-search and than I came back to old query and the module was functional for random count of searching. I don't know how this error works. I think that I need either delete cookies, but I don't know how, or retry after units of time, but I don't know how long is it need wait. – Žanet Feb 12 '20 at 08:22
-1

There is resolution of my problem: https://support.google.com/gsa/answer/4411411#requests

How to disable cookie handling with the Python requests library?

Žanet
  • 64
  • 1
  • 1
  • 7