2

For a project we need to request data through an API (HTTP/1.1), depending on what you find you can then send a request.post with instructions after which the API will send back a response. I made the program multithreaded so that the main program keeps requesting data while in case I want to post an instruction I spawn a thread to do that. (request data only takes 1sec where posting instruction and getting reply can take upto 3sec to respond)

However the problem I am walking into is that sometimes 1 of my threads hangs and only finishes if I issue command thread.join().I can see that it hangs as the data that i get in the main thread should resemble my previous instructions (send by the threads), (I allow a 5second period for the server to resemble the instructions I send prior, so it is not the case that the server is not yet updated). If I would now send the same instructions again I will find that now both instructions make it to the server (the hanging one, and the newly issued one). So somehow sending the new instructions has as a side-effect the previous instructions also get send.

The problem looks related to threading as my code doesn't hang when just executed serial. Looking at posts like this didn't help as I do not know in advance what my instruction needs to be for my asynchronous requests. It is important that I make use of persistent connections and reuse them as that saves alot of time on handshakes etc.

My questions:

  • What is a proper way of handling a connectionpool of persistent connections in a multithreaded way. (so it doesn't hang)
  • How can I debug/troubleshoot the Thread to find out where it hangs.
  • Requests gets recommended as a package often but maybe there are others, better suited for this kind of application?

Example Code:

import requests
from threading import Thread

req = requests.Session()
adapter = requests.adapters.HTTPAdapter(pool_connections = 10, pool_maxsize=10)
req.mount('',adapter)

url='http://python-requests.org'

def main(url=''):
    thread_list=[]
    counter=0
    while True:
        resp = req.get(url)
        interesting = 1 #
        if interesting:
            instructions = {}
            a = Thread(target = send_instructions, kwargs = dict(url = url, instructions = instructions))
            a.start()
            thread_list.append(a)
        tmp=[]
        for x in thread_list:
            if x.isAlive():
                tmp.append(x)
        thread_list = tmp
        if counter>10:
            break
        counter+=1

def send_instructions(url='', instructions=''):
    resp=req.post(url, headers = instructions)
    print(resp)

main(url)
Community
  • 1
  • 1
Wheelster
  • 31
  • 2
  • What do you mean by "1 of my threads hangs"? What do you observe? – Thomas Orozco Nov 13 '15 at 10:27
  • It hangs in the following way. When i get data in the main thread it tells my what my instructions have been in the past. I can see that somehow my instructions haven't made it to the server. So I thread.join() the hanging one and then after 2/3seconds I see that my instructions made it to the server. (EDITED OP) – Wheelster Nov 13 '15 at 12:10
  • What is the purpose of your `for x in thread_list ... join(0)`? Are you intending for each thread to wait for the previous one to finish? – Sanjay Manohar Nov 13 '15 at 12:23
  • my bad. That piece of code was meant to keep track of still active threads. (edited OP) – Wheelster Nov 13 '15 at 12:56
  • Did you ever figure this out? I seem to have something similar with Requests, but only when using https. – Halfgaar May 26 '17 at 13:11

0 Answers0