I'm building a threaded application. I have millions of jobs that I want to queue. can I just laod them all into a queue and then hand them out to threads at around 100 at a time?
Asked
Active
Viewed 88 times
0
-
[What have you tried?](http://whathaveyoutried.com) Have you done any testing? – thegrinner Dec 10 '12 at 19:21
-
I'm just not entirely sure this is a safe way to do it. It seems as though the more I load into the queue, the more likely my threads are to fail but I can't get my head around why. – user1131308 Dec 10 '12 at 19:29
-
If you have some test data to try it on, I recommend you do so. If it ends up like you expect you're golden. – thegrinner Dec 10 '12 at 19:31
-
@user1131308: You could spawn a fixed number of workers and just have an input thread that gives tasks to only free workers. That way, you aren't loading unnecessary data. – Blender Dec 10 '12 at 19:50
-
I think you might find the question [_Python thread pool similar to the multiprocessing Pool?_](http://stackoverflow.com/q/3033952/355230) interesting. – martineau Dec 10 '12 at 20:36