-1

Suppose you want to call some API (open without any KEY needed) which has some rate limit per second (10 requests max per second). Suppose now, I am calling a function (with multiprocessing.Pool) which request data from API and process it. Is there some way to switch ip in order to not get blocked? Maybe a list of ip/ proxies. Can somebody tell me a way to get this done?

Thanks!

BloomShell
  • 833
  • 1
  • 5
  • 20
  • What you *should* be asking is "How can I use multithreading to ensure that I am able to issue requests to a particular site at the rate of 10 requests per second *but no greater*?" Note that this is a task that is probably better suited for for multithreading. – Booboo Mar 05 '22 at 20:32
  • 1
    @Booboo I would have asked that if I was not able to do it. Anyway, I think you mean to use multithreading and a function with a decorator which ensure the call rate limit but this is not inline with my question. – BloomShell Mar 05 '22 at 21:05
  • 1
    I know what you *wanted* to do. What I said you *should* do was my way of subtly saying that what you were actually asking to do is something that most of us would not be comfortable in helping with. But I see now that I should have been more direct. – Booboo Mar 05 '22 at 21:19
  • 1
    What I don't understand is why a person should answer a question with themes unrelated to the question. If you're not comfortable with the subject or don't feel like answering, don't do it. All the answers that have been given are completely wrong or off topic. In any case I would like to close the question as I have already found answers in other questions on this same site but by now it is too late. – BloomShell Mar 05 '22 at 21:29
  • https://stackoverflow.com/questions/56733397/how-i-can-get-new-ip-from-tor-every-requests-in-threads – BloomShell Mar 06 '22 at 09:10

2 Answers2

0

You unfortunately can't retrieve data from an api and do multiprocessing with different ip's because the api will already have your Ip assigned to its request variable. Some Web pages also have http 402 errors which means it is possible to get timed out for sending too many requests.

DoctorRiptide
  • 21
  • 1
0

There certainly are a lot of hacks you can use to get around rate limiting, but you should take a moment and ask yourself 'should I?' You'd likely be violating the terms of service of the service, and in some jurisdictions you could be opening yourself to legal action if you are caught. Additionally, many services implement smart bot detection which can identify bot behavior from request patterns and block multiple IPs. In extreme cases, I've seen teams block the IP range for an entire country to punish botting.

Nick Bailey
  • 3,078
  • 2
  • 11
  • 13
  • My question was mostly motivated by curiosity. I didn't know StackOverflow was a court of law. – BloomShell Mar 05 '22 at 18:17
  • Just looking out for you! If someone's going to do something that could have real world consequences you want to make sure they're going in eyes wide open. – Nick Bailey Mar 05 '22 at 18:29