Python- Killing Threads with SIGINT- global variable not being referenced correctly

Question

I am trying to make it to where my threads can catch a sigint. It looks like to me that kill_received singleton list is in the same namespace of signal_handler() and do_the_uploads() and the same memory location is being referenced. But when I control C when it's running, I see False being printed from "print kill_received[0]" when it should be True since I hit control-C.

kill_received = [False]

def signal_handler(signal, frame):
    global kill_received
    kill_received[0] = True
    print "\nYou pressed Ctrl+C!"
    print (
        "Your logs and their locations are:"
        "\n{}\n{}\n{}".format(debug, error, info))
    sys.exit(0)

def do_the_uploads(file_list, file_quantity,
        retry_list, authenticate):
    """The uploading engine"""
    value = raw_input(
        "\nPlease enter how many conncurent "
        "uploads you want at one time(example: 200)> ")
    value = int(value)
    logger.info('{} conncurent uploads will be used.'.format(value))

    confirm = raw_input(
        "\nProceed to upload files? Enter [Y/y] for yes: ").upper()
    if confirm == "Y":
        kill_received = False
        sys.stdout.write("\x1b[2J\x1b[H")
        q = Queue.Queue()

        def worker():
            global kill_received
            while True and not kill_received[0]:
                print kill_received[0]
                item = q.get()
                upload_file(item, file_quantity, retry_list, authenticate)
                q.task_done()

        for i in range(value):
            t = Thread(target=worker)
            t.setDaemon(True)
            t.start()

        for item in file_list:
            q.put(item)

        q.join()
        print "Finished. Cleaning up processes...",
        #Allowing the threads to cleanup
        time.sleep(4)
        print "done."

From main script:

from modules.upload_actions import do_the_uploads, retry, signal_handler

if __name__ == '__main__':
    signal.signal(signal.SIGINT, signal_handler)

    retry_list = []
    file_list, authenticate, ticket_number = main()
    file_quantity = FileQuantity(len(file_list))
    do_the_uploads(file_list, file_quantity, 
            retry_list, authenticate)

Update:

Still no success, but I changed the syntax to this as it's cleaner:

   def worker():
        global kill_received
        while not kill_received[0]:
            time.sleep(1)
            print kill_received[0]
            item = q.get()
            upload_file(item, file_quantity, retry_list, authenticate)
            q.task_done()

No. When the threads are all compelted then I do...but not during thread execution for all the files they have to upload. — dman, Nov 27 '13 at 21:48
Did you fix the `kill_received = False` line (to `kill_received[0] = False`) and try it again? — Gerrat, Nov 27 '13 at 21:56

score 3 · Accepted Answer · edited May 23 '17 at 11:56

The key to understanding what is going on is the comment you made

No. When the threads are all completed then I do...but not during thread execution for all the files they have to upload.

and this line of code:

q.join()

Contrary to what you are probably expecting, a control-C does NOT cause it to stop waiting for the queue - it doesn't accept the control-C until after this call has returned. So what is happening is that all of your threads have done their jobs and emptied the queue, and then are waiting on the line

item = q.get()

Only after the last thread calls q.task_done does the main thread return and then process the control-C. However, at that point all the threads are stuck waiting for more items from the queue (which they aren't going to get), so they will never exit the loop.

There might be more going on here than this, but to see if this is the problem try a busy wait for the queue to be empty:

while not q.empty():
    time.sleep(0.1)
q.join()

You need the join afterward because the queue being empty means the last upload has been pulled from the queue, not that it has been finished.

One other thing you can add is an item to the queue that signals the thread should finish, such as None. For example,

    def worker():
        global kill_received
        while True and not kill_received[0]:
            print kill_received[0]
            item = q.get()
            if item is None:
                q.task_done()
                break
            upload_file(item, file_quantity, retry_list, authenticate)
            q.task_done()

    for i in range(value):
        t = Thread(target=worker)
        t.setDaemon(True)
        t.start()

    for item in file_list:
        q.put(item)

    for i in range(value):
        q.put(None)

Of course, this assumes that None is not a valid value to upload. This won't help with the control-C issue, but it is something you might find helpful to make sure that the threads exit when the program finishes normally.

As a general help, when you testing things with threads it can be helpful to have a way to print out stack traces for all threads. This SO question talks about how to do that.

Ugh...after many diff. tries there is no winning on this. Is there anyway to make q.join() release if a sigint is recieved? — dman, Nov 29 '13 at 04:18
@dman Did the loop with `time.sleep` not work? If you have enough threads/few enough jobs, then the queue will empty out pretty quick. Did you try having only one or two worker threads with a reasonably large number of jobs? You could also try using `while q.unfinishedtasks:` as the loop condition. — Rob Watts, Dec 02 '13 at 23:07

Gerrat · Answer 2 · 2013-11-27T21:48:58.727

2

The reason you see False printed is because it never gets a chance to print it. You killed it before it every hits your print kill_received[0] statement.

Think about it. There is probably a small chance that you could hit Ctrl-C between execution of this statement:

while True and not kill_received[0]:

and this statement:

print kill_received[0]

but it's improbable. Interrupting any of the threads at any other time will cause them to stop looping (from your while statement), and never print anything.

EDIT: You have the line: kill_received = False which may be causing you issues. It should probably be kill_received[0] = False

edited Nov 27 '13 at 21:48

answered Nov 27 '13 at 21:42

Gerrat

28,863
9
73
101

No, the threads are continuing and printing False as the loop continues. "while True and not kill_received " remains to be True so the threads continue. – dman Nov 27 '13 at 21:44
@dman Try adding the line `print 'Checking: %s' % kill_received[0]` after the while loop to see if they are exiting the loop – Rob Watts Nov 27 '13 at 21:47
@Rob, tried this. I placed it after the while loop but it never runs, so the loop is never exiting. – dman Nov 27 '13 at 21:53

Python- Killing Threads with SIGINT- global variable not being referenced correctly

2 Answers2