Terminante threads when main program exits

Question

I am trying to get started with multithreading in Python. I have multiple threads that acquire a lock, perform an operation, release the lock, and output the result to a csv file. The threads should be terminated in case the main thread finishes (e.g. through pressing Ctrl+C). How is this done?

I thought making the threads daemons should do the job, because "the entire Python program exits when only daemon threads are left" (official Python documentation). This is not sufficient. According to python: how to terminate a thread when main program ends, I then tried catching a KeyboardInterrupt and then terminating the threads manually. Also this is not working, it seems that the KeyboardInterrupt is not catched correctly.

It makes me think, there is something about multithreading in Python that I misunderstood... Find attached my code:

import random
import csv
import sys
from datetime import datetime
import threading

lock = threading.Lock()

def observer(obs):
    print("In " + str(obs))
    sys.stdout.flush()        

    with open("data/test_" + str(obs) + ".csv", 'a', newline='') as csvfile:
        while True:
            datenow = datetime.today()        
            lock.acquire()
            print(str(obs) + "acquired")
            sum = 0
            for i in range(10000):
                sum = sum + random.random()
            print(str(obs) + "released")
            sys.stdout.flush()
            lock.release()

            writer = csv.writer(csvfile, delimiter=',', quoting=csv.QUOTE_ALL)
            writer.writerow([datenow.isoformat(), sum/10000])
            #print(str(obs) + ": " + datenow.isoformat() + " " + str(sum/1000))
            sys.stdout.flush()

if __name__ == "__main__":
    observe = [1, 2, 3, 4]

    processes = []
    for obs in observe:
        process = threading.Thread(target=observer, args=(obs,), daemon=True)
        processes.append(process)

    print("Start processes")
    for p in processes:
        p.start()

    print("Started")
    try:
        for p in processes:
            p.join()
    except KeyboardInterrupt:
        for p in processes:
            p.terminate()
        print("Keyboard interrupt")

    print("Finished")

Thanks!

score 0 · Answer 1 · answered Dec 23 '17 at 19:29

This is wrong

try:
    for p in processes:
        p.join()
except KeyboardInterrupt:
    for p in processes:
        p.terminate()
    print("Keyboard interrupt")

Your threads never exit the while True: loop, so you will be joining them forever. But that's not the point.

If Ctrl-C (aka KeyboardInterrupt) is not being caught my best bet:

You are running Python under Windows
The shell under which you run your script manipulates the processes and you actually never see the Ctrl-C because your program is being terminated abruptly.

You problem would then be:

Your program doesn't actually know it has terminated and that's why the threads hang.

If the assumptions made above are right, try this code (from Python - Windows - Exiting Child Process when "unrelated" parent dies/crashes)

import sys

def win_wait_for_parent(raise_exceptions=False):
    if not sys.platform == 'win32':
        return True

    # When started under cygwin, the parent process will die leaving a child
    # hanging around. The process has to be waited upon
    import ctypes
    from ctypes.wintypes import DWORD, BOOL, HANDLE
    import os
    import threading

    INFINITE = -1
    SYNCHRONIZE = 0x00100000

    kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)

    kernel32.OpenProcess.argtypes = (DWORD, BOOL, DWORD)
    kernel32.OpenProcess.restype = HANDLE

    kernel32.WaitForSingleObject.argtypes = (HANDLE, DWORD)
    kernel32.WaitForSingleObject.restype = DWORD

    phandle = kernel32.OpenProcess(SYNCHRONIZE, 0, os.getppid())

    def check_parent():
        # Get a token with right access to parent and wait for it to be
        # signaled (die). Exit ourselves then
            kernel32.WaitForSingleObject(phandle, INFINITE)
            os._exit(0)

    if not phandle:
        if raise_exceptions:
            raise ctypes.WinError(ctypes.get_last_error())

        return False

Thanks for the answer! I am not 100% sure what the function win_wait_for_parent() does exactly and, thus, how to include it in my program. Does it simply return True if there is no parent anymore? — TheBirdIsTheWord, Dec 23 '17 at 20:50
The function returns always because it does either wait for a parent to finish (waits in a background thread) and it will then exit your process with `os._exit(0)` or returns immediately if there is no parent. Just *call* the function. — mementum, Dec 24 '17 at 22:40

Terminante threads when main program exits

1 Answers1