2

EDIT So my B class receives messages for a telegram bot, this is the code, for each message received call the function in class A

@bot.message_handler(commands=['get'])
def get(message):
    uid = message.chat.id
    #url that should return a 404 error
    response = get_response(url)
    #if I get a "get" command while the previous message received is still trying to execute the requests
    #the requests do not return a 404 message as it should (I know this because I deliberately pass a link that returns an error 404)
    if response.status_code == 404:
        bot.send_message(uid,"404")
        return
    else:
        #other

this is the code of the class A function

def get_response(url):
ua = UserAgent()
while (True):

    for i in range(1, len(proxies)):
        proxy = next(proxy_pool)
        try:
            header = {'User-Agent': str(ua.random)}
            response = requests.get(url, headers=header, proxies={"http": proxy, "https": proxy}, verify=False,
                                    timeout=10)
            return response
        except:
           print("Skipping. Connnection error")

    #other conditions to break the while loop

return None

This method is called by other classes B and C and by different threads, the problem is that I think the requests.get() is not a threadsafe function and therefore does not work well when it is called by multiple threads at the same time. For example if I call requests.get() simultaneously from multiple threads with links that should return a 404 error, this error is not returned, while it is returned if the call to requests.get() is made individually. So how to make this method thread safe and work properly if it is called by multiple threads?

JayJona
  • 469
  • 1
  • 16
  • 41
  • I don't see any reason that `requests.get` shouldn't be thread safe. For starters, with the GIL only one thread will be in the code at a time. Most of its execution time is going to be blocking on I/O. Are you sure it's not an issue having to do with the sites you're requesting? If you're sharing a session amongst threads, [there is cause for concern](https://stackoverflow.com/questions/18188044/is-the-session-object-from-pythons-requests-library-thread-safe) but that's not the case here. – ggorlen Dec 12 '19 at 01:15
  • A [mcve] would be nice. – ggorlen Dec 12 '19 at 01:23
  • Have you see grequests ? https://github.com/spyoungtech/grequests – GiovaniSalazar Dec 12 '19 at 01:30
  • @ggorlen code updated – JayJona Dec 12 '19 at 01:43
  • @GiovaniSalazar I need to use proxy, timeout and headers, does grequests.get() support them – JayJona Dec 12 '19 at 01:43
  • 1
    Thanks, although this still isn't really reproducible unfortunately. I should be able to copy+paste, hit run, see the 404s. – ggorlen Dec 12 '19 at 01:45
  • @james I think yes .. https://gist.github.com/windj007/d49391fccc1cf6e22856ebaabf892fc6 – GiovaniSalazar Dec 12 '19 at 01:49

0 Answers0