Trying to fix tkinter GUI freeze-ups (using threads)

Question

I have a Python 3.x report creator that is so I/O bound (due to SQL, not python) that the main window will "lock up" for minutes while the reports are being created.

All that is needed is the ability to use the standard window actions (move, resize/minimize, close, etc.) while the GUI is locked-up (everything else on the GUI can stay "frozen" until all reports have finished).

Added 20181129 : In other words, tkinter must only control the CONTENTS of the application window and leave handling of all standard (external) window controls to the O/S. If I can do that my problem disappears and I don't need to use threads/subprocesses all (the freezeups become acceptable behaviour similar in effect to disabling the "Do Reports" button).

What is the easiest/simplest way (= minimum disturbance to existing code) of doing this - ideally in a way that works with Python >= 3.2.2 and in a cross-platform way (i.e. works on at least Windows & linux).

---

Everything below is supporting information that explains the issue in greater detail, approaches tried, and some subtle issues encountered.

Things to consider:

• Users choose their reports then push a "Create Reports" button on the main window (when the real work starts and the freezeup occurs). Once all reports are done the report creation code displays a (Toplevel) "Done" window. Closing this window enables everything in the main window, allowing users to exit the program or create more reports.

• Added 20181129: At apparently random intervals (several seconds apart) I can move the window.

• Except for displaying the "Done" window, the report creation code does not involve the GUI or tkinter in any way.

• Some data produced by the report creation code must appear on the "Done" window.

• There's no reason to "parallelize" report creation especially since the same SQL server & database is used to create all reports.

• In case it affects the solution : I'll eventually need to display the report names (now shown on the console) on the GUI as each report gets created.

• First time doing threading/subprocessing with python but am familiar with both from other languages.

• Added 20181129 : Development environment is 64 bit Python 3.6.4 on Win 10 using Eclipse Oxygen (pydev plugin). Application must be portable to at least linux.

---

The simplest answer seems to be to use threads. Only one additional thread is needed (the one that creates the reports). The affected line:

DoChosenReports()  # creates all reports (and the "Done" window)

when changed to:

from threading import Thread

CreateReportsThread = Thread( target = DoChosenReports )
CreateReportsThread.start()
CreateReportsThread.join()  # 20181130: line omitted in original post, comment out to unfreeze GUI 

successfully produces the reports with their names being displayed on the console as they get created.
However, the GUI remains frozen and the "Done" window (now invoked by the new thread) never appears. This leaves the user in limbo, unable to do anything and wondering what, if anything, has happened (which is why I want to display the filenames on the GUI as they get created).

BTW, After the reports are done the report creation thread must quietly commit suicide before (or after) the Done window is shown.

I also tried using

from multiprocessing import Process
    
ReportCreationProcess = Process( target = DoChosenReports )
ReportCreationProcess.start()

but that fell afoul of the main programs "if (_name_ == '_main_) :' " test.

---

Added 20181129 : Just discovered the wait_variable() universal widget method). Basic idea is to launch the create report code as an do-forever thread (daemon?) controlled by this method (with execution controlled by the Do reports button in the GUI).

---

From web research I know that all tkinter actions should be made from within the main (parent) thread, meaning that I must move the "Done" window to that thread.
I also need that window to display some data (three strings) it receives from the "child" thread. I'm thinking of using use application-level globals as semaphores (only written to by the create report thread and only read by the main program) to pass the data. I'm aware that this can be risky with more than two threads but doing anything more (e.g. using queues?) for my simple situation seems like overkill.

---

To summarize: What's the easiest way to allow the user to manipulate (move, resize, minimize, etc.) an application's main window while the window is frozen for any reason. In other words, the O/S, not tkinter, must control the frame (outside) of the main window.
The answer needs to work on python 3.2.2+ in a cross-platform way (on at least Windows & linux)

Answer 1

You'll need two functions: the first encapsulates your program's long-running work, and the second creates a thread that handles the first function. If you need the thread to stop immediately if the user closes the program while the thread is still running (not recommended), use the daemon flag or look into Event objects. If you don't want the user to be able to call the function again before it's finished, disable the button when it starts and then set the button back to normal at the end.

import threading
import tkinter as tk
import time

class App:
    def __init__(self, parent):
        self.button = tk.Button(parent, text='init', command=self.begin)
        self.button.pack()
    def func(self):
        '''long-running work'''
        self.button.config(text='func')
        time.sleep(1)
        self.button.config(text='continue')
        time.sleep(1)
        self.button.config(text='done')
        self.button.config(state=tk.NORMAL)
    def begin(self):
        '''start a thread and connect it to func'''
        self.button.config(state=tk.DISABLED)
        threading.Thread(target=self.func, daemon=True).start()

if __name__ == '__main__':
    root = tk.Tk()
    app = App(root)
    root.mainloop()

Answer 2

I found a good example similar to what you want to do in one of the books I have which I think shows a good way of using threads with tkinter. It's Recipe 9.6 for Combining Tkinter and Asynchronous I/O with Threads in the first edition of the book Python Cookbook by Alex Martinelli and David Ascher. The code was written for Python 2.x, but required only minor modifications to work in Python 3.

As I said in a comment, you need to keep the GUI eventloop running if you want to be able to interact with it or just to resize or move the window. The sample code below does this by using a Queue to pass data from the background processing thread to the main GUI thread.

Tkinter has a universal function called after() which can be used schedule a function to be called after certain amount time has passed. In the code below there's a method named periodic_call() which processes any data in the queue and then calls after() to schedule another call to itself after a short delay so the queue data processing will continue.

Since after() is part of tkinter, it allows the mainloop() to continue running which keeps the GUI "alive" between these periodic queue checks. It can also make tkinter calls to update the GUI if desired (unlike code that's running in separate threads).

from itertools import count
import sys
import tkinter as tk
import tkinter.messagebox as tkMessageBox
import threading
import time
from random import randint
import queue

# Based on example Dialog 
# http://effbot.org/tkinterbook/tkinter-dialog-windows.htm
class InfoMessage(tk.Toplevel):
    def __init__(self, parent, info, title=None, modal=True):
        tk.Toplevel.__init__(self, parent)
        self.transient(parent)
        if title:
            self.title(title)
        self.parent = parent

        body = tk.Frame(self)
        self.initial_focus = self.body(body, info)
        body.pack(padx=5, pady=5)

        self.buttonbox()

        if modal:
            self.grab_set()

        if not self.initial_focus:
            self.initial_focus = self
        self.protocol("WM_DELETE_WINDOW", self.cancel)
        self.geometry("+%d+%d" % (parent.winfo_rootx()+50, parent.winfo_rooty()+50))
        self.initial_focus.focus_set()

        if modal:
            self.wait_window(self)  # Wait until this window is destroyed.

    def body(self, parent, info):
        label = tk.Label(parent, text=info)
        label.pack()
        return label  # Initial focus.

    def buttonbox(self):
        box = tk.Frame(self)
        w = tk.Button(box, text="OK", width=10, command=self.ok, default=tk.ACTIVE)
        w.pack(side=tk.LEFT, padx=5, pady=5)
        self.bind("<Return>", self.ok)
        box.pack()

    def ok(self, event=None):
        self.withdraw()
        self.update_idletasks()
        self.cancel()

    def cancel(self, event=None):
        # Put focus back to the parent window.
        self.parent.focus_set()
        self.destroy()


class GuiPart:
    TIME_INTERVAL = 0.1

    def __init__(self, master, queue, end_command):
        self.queue = queue
        self.master = master
        console = tk.Button(master, text='Done', command=end_command)
        console.pack(expand=True)
        self.update_gui()  # Start periodic GUI updating.

    def update_gui(self):
        try:
            self.master.update_idletasks()
            threading.Timer(self.TIME_INTERVAL, self.update_gui).start()
        except RuntimeError:  # mainloop no longer running.
            pass

    def process_incoming(self):
        """ Handle all messages currently in the queue. """
        while self.queue.qsize():
            try:
                info = self.queue.get_nowait()
                InfoMessage(self.master, info, "Status", modal=False)
            except queue.Empty:  # Shouldn't happen.
                pass


class ThreadedClient:
    """ Launch the main part of the GUI and the worker thread. periodic_call()
        and end_application() could reside in the GUI part, but putting them
        here means all the thread controls are in a single place.
    """
    def __init__(self, master):
        self.master = master
        self.count = count(start=1)
        self.queue = queue.Queue()

        # Set up the GUI part.
        self.gui = GuiPart(master, self.queue, self.end_application)

        # Set up the background processing thread.
        self.running = True
        self.thread = threading.Thread(target=self.workerthread)
        self.thread.start()

        # Start periodic checking of the queue.
        self.periodic_call(200)  # Every 200 ms.

    def periodic_call(self, delay):
        """ Every delay ms process everything new in the queue. """
        self.gui.process_incoming()
        if not self.running:
            sys.exit(1)
        self.master.after(delay, self.periodic_call, delay)

    # Runs in separate thread - NO tkinter calls allowed.
    def workerthread(self):
        while self.running:
            time.sleep(randint(1, 10))  # Time-consuming processing.
            count = next(self.count)
            info = 'Report #{} created'.format(count)
            self.queue.put(info)

    def end_application(self):
        self.running = False  # Stop queue checking.
        self.master.quit()


if __name__ == '__main__':  # Needed to support multiprocessing.
    root = tk.Tk()
    root.title('Report Generator')
    root.minsize(300, 100)
    client = ThreadedClient(root)
    root.mainloop()  # Display application window and start tkinter event loop.

Answer 3

I've modified the question to include the accidentally omitted but critical line. The answer to avoiding GUI freezeups turns out to be embarrassingly simple:

Don't call ".join()" after launching the thread.

In addition to the above, a complete solution involves:

• Disabling the "Do Reports" button until the "create report" thread finishes (technically not necessary but preventing extra report creation threads also prevents user confusion);
• Having the "create report" thread update the main thread using these events:
  • "Completed report X" (an enhancement that displays progress on GUI), and
  • "Completed all reports" (display the "Done" window and reenable the "Do Reports" button);
• Moving the invocation of the "Done" window to the main thread, invoked by the above event; and
• Passing data with the event instead of using shared global variables.

A simple approach using the multiprocessing.dummy module (available since 3.0 and 2.6) is:

    from multiprocessing.dummy import Process

    ReportCreationProcess = Process( target = DoChosenReports )
    ReportCreationProcess.start()

again, note the absence of a .join() line.

As a temporary hack the "Done" window is still being created by the create report thread just before it exits. That works but does cause this runtime error:

RuntimeError: Calling Tcl from different appartment  

however the error doesn't seem to cause problems. And, as other questions have pointed out, the error can be eliminated by moving the creation of the "DONE" window into the main thread (and have the create reports thread send an event to "kick off" that window).

Finally my thanks to @TigerhawkT3 (who posted a good overview of the approach I'm taking) and @martineau who covered how to handle the more general case and included a reference to what looks like a useful resource. Both answers are worth reading.