python threading running process in backend

Question

I was trying to make some text report file from some data source which takes enormous time and to simulate this I wrote the following code

I planned to do it using thread and thought t.daemon = True would solve the purpose, but the program doesn't exit till the operation is complete

import random
import threading
import time
import logging

logging.basicConfig(level=logging.DEBUG,
                    format='(%(threadName)-10s) %(message)s',
                    )


def worker():
    """thread worker function"""
    t = threading.currentThread()
    tag = random.randint(1, 64)
    file_name = "/tmp/t-%d.txt" % (tag)
    logging.debug('started writing file - %s', file_name)
    f = open(file_name, 'w')
    for x in xrange(2 ** tag):  # total no of lines is 2**tag
        f.write("%d\n" % x)
    logging.debug('ending')
    f.close()
    return

# to simulate 5 files
for i in range(5):
    t = threading.Thread(target=worker)
    t.daemon = True
    t.start()

main_thread = threading.currentThread()
for t in threading.enumerate():
    if t is main_thread:
        continue
    logging.debug('joining %s', t.getName())
    t.join()

When I removed t.join() then some of the data written till program exits and the program exits quickly, but adding t.join() keeps program running till end. Is there any way to exit from program but the process should still be running to complete the task in backend.

Answer 1

You aren't looking for a daemon. In fact you want to make sure your process isn't a daemon because it will get killed once that's all that's left and your program exists. You are looking to detach your thread.

Note: lowered max to 28 in case I forgot to kill processes (and so it won't take my entire disk). You will need to kill each process individually if you want them to stop! ie "kill 13345" if you had the message "exiting main 13345" (where that thread is over 2**25)

Also note: thread joining will keep going until the end because your program is not done running and is waiting to join the threads.

Here's what you want:

import logging 
import random
import multiprocessing
import time
import sys


#Make sure you don't write to stdout after this program stopped running and sub-processes are logging!
logging.basicConfig(level=logging.DEBUG,
                    format='(%(threadName)-10s) %(message)s',
                    )

def detach():
    p = multiprocessing.current_process()
    name = "worker" + str(p.pid)
    cc = multiprocessing.Process(name=name, target=worker)
    cc.daemon = False
    cc.start()
    logging.debug('Detached process: %s %s', p.name, p.pid)
    sys.stdout.flush()

def worker():
    """thread worker function"""
    #Should probably make sure there isn't already a thread processing this file already...
    tag = random.randint(5, 33) #Stop at 33 to make sure we don't take over the harddrive (8GB)
    file_name = "/tmp/t-%d.txt" % (tag)
    if tag > 26:
      logging.warning('\n\nThe detached process resulting from this may need to be killed by hand.\n')
    logging.debug('started writing file - %s', file_name)
    #Changed your code to use "with", available in any recent python version
    with open(file_name, 'w') as f:
        for x in xrange(2 ** tag):  # total no of lines is 2**tag
            f.write("%d\n" % x)
    return
#Stackoverflow: Keep scrolling to see more code!

# to simulate 5 files
for i in range(5):
    t = multiprocessing.Process(target=detach)
    t.daemon = False
    t.start()
    time.sleep(0.5)
    t.terminate()
    logging.debug("Terminating main program")