3
for message in messages:
                self.numberT+=1
                if(self.numberT%100==0):
                    print str(self.numberT)
                thread1= threading.Thread(target=self.worker_gender,args=(message,))
                thread2 = threading.Thread(target=self.worker_sentiment,args=(message,))

                thread1.start()
                thread2.start()

So I start my code out like this, simply opening up two threads, give two separate functions and throw both of them the same input. In both functions I call APIs

response = unirest.get(correct stuff),

I then do what I want with the response. I think I'm doing some unsafe threading here. The response from both are JSON files, with obviously different fields. I find errors being caught that response is being used before defined. The response variable is a dictionary, so I can access it as such, when I do it I find fields dont exist(sometimes). I think my threaded code is being executed on the wrong response, I get response that I would expect from thread 1, but it is being used in thread 2 and so on. If someone could point me to some resources for making that not happen that would be great.

Another possible issue is for each message I open 2 threads that are accessing two apis. So when this starts going I can have 1000 threads for example calling API 1 and 1000 threads calling API 2 in parallel . I don't know if that would have something to do with it too.

I really think I'm creating artificial race conditions with http requests somehow.

Thank you for the help fellas.

Eigenvalue
  • 1,093
  • 1
  • 14
  • 35
  • Is unirest thread-safe? Otherwise I recommend trying Requests (http://docs.python-requests.org/en/latest/). – Pieter van den Ham Jul 04 '15 at 01:26
  • @PietervandenHam Unirest seems to be the same as requests. – Malik Brahimi Jul 04 '15 at 01:53
  • 1. Don't use 1000 threads, to make 1000 requests. [Use a thread pool](http://stackoverflow.com/a/23284285/4279) or async.io instead 2. To see whether your code uses data that it shouldn't, test with multiple processes (just change imports (e.g., remove `.dummy`), the API is the same) – jfs Jul 04 '15 at 02:42
  • Thank you for the responses everybody. – Eigenvalue Jul 04 '15 at 04:06

0 Answers0