Threads are not joined properly in Idle

Question

I am trying to create a work and a thread pool. Each thread from the pool tries to pick a work untill work is empty. Then all the threads join. When I run the program in idle, it seems that threads are not joining properly (Sometimes it executes properly). Even sometimes all threads do not participate in picking work properly.

Edit:But when I run the same in command shell, it works fine.

Below is the related code.

import threading
import time

threads = []
work = range(1000)
lock = threading.Lock()

def f():
    global work
    th = threading.currentThread()
    name = str(th.getName())
    while True:
        lock.acquire()
        try:
            size = str(len(work))
##            print "\n Inside  "+ name +" "+ size + "\n"
            if size!="0":
                w = work.pop()
            else:
                break
        finally:
            lock.release()
        print "\n"+ name +" "+ size + "\n"



start1 = time.time()
for i in range(10):
    t = threading.Thread(target = f)
    threads.append(t)
    t.start()

for th in threads:
    print "joining"
    th.join()

end1 = time.time()

start2 = time.time()

work = range(1000)
for i in work:
    print i

end2 = time.time()

print end1-start1, end2-start2

Python 2.7.9 on Win 7. I presume the symptom of 'not joining properly' is that nothing is printed after 'joining'. After this happened once, I got several successful runs I do notice that the print text and returns get seprated. I do not see the point of printing 0 to 9999 in separate lines. PS. `global work` is not needed since you mutate but do not rebind `work`. — Terry Jan Reedy, Mar 29 '15 at 23:31

score 0 · Answer 1 · edited May 23 '17 at 11:50

How do you know that each thread is not acquiring work from the pool? Or that all threads are not joining.

What I would guess is the case is that some threads are fast runners and drain the pool before some of the slow runners can get a chance to draw any work from the pool and it appears they didn't do anything thanks to your friendly scheduler. I was able to run this successfully through several hundred iterations and it appears that all threads joined successfully every time. Tested by modifying using python 2.7.8:

i=0
for th in threads:
    th.join() #blocks until successful
    i+=1
assert i == 10

Further more increase or decreasing the thread count, pool size or number of iterations produces the same results. Unless there is more code you've cut out this code is correct as far as I can see.

EDIT: Concerning IDLE - From what I can tell IDLE has issues a plenty when running threaded applications. This thread makes mention of them but I cannot find any specific bugs or references about why this is the case. It's most likely a race condition in IDLE causing threads to not run/look like they aren't running/etc.

Also I am able to execute this code (with my modification) in IDLE on my system, the assert is passed and the time taken printed without problem. If your problem is related to what you're seeing printed on the console in IDLE it is important to consider that IDLE's print function is very likely not thread-safe. You may not see all the output on every execution because IDLE is flushing the buffer and not writing it (or writing over it before printing).

The problem with the linked question https://stackoverflow.com/questions/3153792/python-idle-compatible-with-multithreading was simply a matter of the minimum value of threading.activeCount being 2 instead of 1 when running under Idle while the code looped until the count dropped to 1, which cannot happen. I posted a new answer there with a fix. — Terry Jan Reedy, Mar 29 '15 at 23:31

Threads are not joined properly in Idle

1 Answers1