i would like to understand how to use python threading and queue. my goal is to have 40 threads always alive, this is my code:
for iteration iterations: # main iteration
dance = 1
if threads >= len(MAX_VALUE_ITERATION) :
threads = len(MAX_VALUE_ITERATION)-1 # adjust number of threads as because in this iteration i have just x subvalues
else:
threads = threads_saved # recover the settings or the passed argument
while dance <= 5: # iterate from 1 to 5
request = 0
for lol in MAX_LOL: # lol iterate
for thread_n in range(threads): # MAX threads
t = threading.Thread(target=do_something)
t.setDaemon(True)
t.start()
request += 1
main_thread = threading.currentThread()
for t in threading.enumerate():
if t is main_thread:
continue
if request < len(MAX_LOL)-1 and settings.errors_count <= MAX_ERR_COUNT:
t.join()
dance += 1
The code you see here it was cleaned because it was long to debug for you, so i try to semplify a little bit. As you can see there are many iteration , i start from a dbquery and i fetch the result in the list (iterations) next i adjust the max number of the threads allowed then i iterate again from 1 to 5 (it's an argoument passed to the small thread) then inside the value fetched from the query iteration there is a json that contain another list i need to iterate again ... and then finally i open the threads with start and join ...
The script open x threads and then, when they (all or almost) finish it will open others threads ... but my goal is to keep max X threads forever , i mean once one thread is finished another have to spawn and so on ... until the max_number of the threads is reached.
i hope you can help me. Thanks