0

I'm having some troubles with python threads. I'm writing a software package that plots data received from multiple devices. I have a plot thread that plots the data once it has received a set of data from all devices, and a data retrieval thread for each device. The application plots data continuously (as fast as data can be retrieved from the device) until the user hits a button. I have a threading.Event() self.stop_thread that is checked frequently to back out of the threaded loops. The threads hit the check, break out of the loop, but are still 'running' according to my debugger and threading.active_count(). Does anyone know why this is happening and how can I get it to stop? I need to know these threads are gone before I move on to another function of the application. The following three methods are where the issues arise. 

# initalizes startup settings, starts a thread to carry out
# plotting and a seperate thread to carry out data retrieval
def start_plot_threads(self):
    if not self.abstraction.connected:
        self.connect_to_device()
        if not self.abstraction.connected:
            return
    self.stop_thread.clear()
    self.pause_thread.clear()
    for device in self.devices:
        device.pause_thread.clear()
        device.stop_thread.clear()
        device.change_units.set()
    self.presentation.enable_derivative()
    self.presentation.show_average_button.SetValue(False)
    self.presentation.show_average_button.Disable()
    self.abstraction.multi_plot_data = {}
    try:
        if self.plot_thread.is_alive():
            return
    except Exception:
        pass
    self.plot_thread = Thread(target=self.plot_data)
    self.plot_thread.daemon = True
    self.plot_thread.start()
    for device in self.devices:
        thread = Thread(target=self.retrieve_data,
                        kwargs={'device': device},
                        name="Data Retrieval Thread %s" % device.instr_id)
        thread.daemon = True
        thread.start()

# waits for plot data to be thrown on a thread safe queue by the data
# retrieval thread and plots it. data comes in as a tuple of the form
# (y_data, label, x_data)
def plot_data(self):
    multiplot = False
    if len(self.devices) > 1:
        multiplot = True
        plot_data = []
    while not self.stop_thread.is_set():
        try:
            data = self.plot_data_queue.get()
        except Empty:
            pass
        else:
            if multiplot:
                scan = {}
                scan['y_data'] = [data[0]]
                scan['labels'] = [data[1]]
                scan['x_data'] = data[2]
                plot_data.append(scan)
                if len(plot_data) == len(self.devices):
                    self.presentation.plot_multiline(plot_data, average=False)
                    self.abstraction.multi_plot_data = plot_data
                    plot_data = []
            else:
                self.presentation.plot_signal(data[0], data[1])

# the intent is that the data retrieval thread stays in this loop while
# taking continuous readings
def retrieve_data(self, device):
    while True:
        if device.stop_thread.is_set():
            return
        while device.pause_thread.is_set():
            if device.stop_thread.is_set():
                return
            sleep(0.1)
        y = self.get_active_signal_data(device)
        if not y:
            return
        self.plot_data_queue.put(
            (y, device.name, device.x_data))
        self.abstraction.y_data = [y]
        try:
            self.update_spectrum(device)
        except DeviceCommunicationError, data:
            self.presentation.give_connection_error(data)
        self.presentation.integ_time = device.prev_integ

I apologize for the extra bulk in the methods. They are straight from my code base.

jgramse
  • 193
  • 10
  • Why do you set `daemon = True` for your threads? – Vasiliy Faronov Jun 10 '15 at 00:21
  • I set daemon = True so that the application will exit when the main thread is terminated. I'm not worried about data loss or whether or not the threads have finished their task. I just need to make sure the threads are terminated before moving on to another function of the application. – jgramse Jun 10 '15 at 16:22

1 Answers1

0

The reason why your threads continue running is unknown- device.stop_thread.is_set(): (What is doing the setting??)

However you can guarantee that all your threads have stopped by retaining a handler on each thread ( by appending each thread object to a list) and once you have started all your threads you can then proceed to thread.join() them.

threads = []
for job in batch:
     thr = threading.Thread(target=do_job, args = (job))
     thr.start()
     threads.append(thr)
#join all the threads
for thr in threads:
    thr.join()

Join will wait for the thread to complete before moving on.

Python Docs: https://docs.python.org/2/library/threading.html

John
  • 2,410
  • 1
  • 19
  • 33
  • stop_thread.set() is called when the user clicks one of many buttons in the application. It is being set correctly, checked correctly, and the execution of the target method is being returned from correctly. I have tried thread.join() (I had the threads saved in a list at one point) but it just stays at that blocking call to join() forever. The threads are not executing any task as they have already left their target methods. They are just running... forever... and I don't know why. – jgramse Jun 09 '15 at 23:19
  • thread.join() will do what you expect, so you should go back to that code. I'm going to bet your problem lies with http://stackoverflow.com/questions/1294382/what-is-a-global-interpreter-lock-gil essentially what you need to know is that if you don't have and execution breaks in your code other threads will be COMPLETELY (or very nearly) blocked from running. Implement some time.sleep if you are in a read loop somewhere else- this will give other threads a chance to run. – John Jun 09 '15 at 23:25
  • Thanks John. Not sure why it wasn't working before but thread.join() seems to be working now. – jgramse Jun 10 '15 at 16:51