0

In Python 2.7.11. I have a tkinter GUI where the user enters files into a Listbox. The RUN button is supposed to read each file and insert data into a database. I need the reading to be in series, not parallel. I want to run the reading in a separate process. I ran the following code outside of tkinter, as a test, and I got the desired results... The main thread appears to finish while the reader is still chugging along in the background.

import multiprocessing as mp
import time
import F06


def _worker(li):
    for path in li:
    print('worker is processing:={}'.format(path))
    reader = F06.Reader()
    reader.read_sol_106(path)
    time.sleep(0.5)
    print('process complete')

if __name__ == '__main__':

    # make a list of files to read
    li = [
    'Results/1201301__SOL106.f06',
    'Results/1201302__SOL106.f06',
    'Results/1201303__SOL106.f06',
    ]
    p = mp.Process(target=_worker, args=(li,))
    p.start()
    print('main thread finished')

I tried adding the code in a similar fashion to my tkinter program and I cannot get it to run. The tkinter GUI is in a class, not sure if this is contributing to the problem, just FYI. This is the following error feedback:

Exception in Tkinter callback
Traceback (most recent call last):
  File "C:\Python27\lib\lib-tk\Tkinter.py", line 1537, in __call__
    return self.func(*args)
  File "C:\Users\00835182\Documents\METHODS\01_PET_PROJECTS\13_Py_SQL\DB_GUI.py", line 122, in run
    p.start()
  File "C:\Python27\lib\multiprocessing\process.py", line 130, in start
    self._popen = Popen(self)
  File "C:\Python27\lib\multiprocessing\forking.py", line 277, in __init__
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
  File "C:\Python27\lib\multiprocessing\forking.py", line 199, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "C:\Python27\lib\pickle.py", line 224, in dump
    self.save(obj)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 425, in save_reduce
    save(state)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 687, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\multiprocessing\forking.py", line 67, in dispatcher
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 401, in save_reduce
    save(args)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 554, in save_tuple
    save(element)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 425, in save_reduce
    save(state)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 687, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 731, in save_inst
    save(stuff)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 687, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 731, in save_inst
    save(stuff)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 687, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 731, in save_inst
    save(stuff)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 687, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 687, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 731, in save_inst
    save(stuff)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 687, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 313, in save
    (t.__name__, obj))
PicklingError: Can't pickle 'tkapp' object: <tkapp object at 0x024932F0>
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Python27\lib\multiprocessing\forking.py", line 381, in main
    self = load(from_parent)
  File "C:\Python27\lib\pickle.py", line 1384, in load
    return Unpickler(file).load()
  File "C:\Python27\lib\pickle.py", line 864, in load
    dispatch[key](self)
  File "C:\Python27\lib\pickle.py", line 886, in load_eof
    raise EOFError
EOFError

I am not sure, but the problem my lie in the 'self' argument. My code, at least what I think are the relevant portions are:

def run(self):  # ... linked to "Run" button 'command'
    f06_files = self.frame_2_listbox.get(0, 'end')  # ............. get List of results files
    p = multiprocessing.Process(target=self._run, args=(f06_files,))
    # this also did not work
    # p = multiprocessing.Process(target=self._run, args=(None, f06_files))
    p.start()

def _run(self, li):
    num_files = len(li)  # ........................................ total number of file
    file_counter = 0  # ........................................... file counter

    for path in li:  # ............................................ iterate over files

        file_counter += 1  # ...................................... increment file counter
        s = 'File  {}  of  {}'.format(file_counter, num_files)  # . string msg to user
        print(s)

        # read the results file
        reader = Reader()  # ...................................... instantiate Reader, clears for reuse
        reader.read_sol_106(path)  # .............................. read results file, extract loads

        # retrieve loads
        bars = reader.get_bar_forces()  # ......................... List of Tuples
        beams = reader.get_beam_forces()  # .......................
        bushs = reader.get_bush_forces()  # .......................
        cons = reader.get_con_forces()  # .........................
        rods = reader.get_rod_forces()  # .........................
        shells = reader.get_shell_forces()  # .....................
        gaps = reader.get_gap_forces()  # .........................
        springs = reader.get_spring_forces()  # ...................

        # insert loads into database
        self.insert_data(file=path, bars=bars, beams=beams, bushs=bushs, cons=cons, rods=rods, shells=shells, gaps=gaps, springs=springs)

I'm ready to learn whatever you can teach me.

twegner
  • 443
  • 1
  • 5
  • 21
  • this is because multiprocessing happens through pickling objects and passing them to other processes. I bet that you don't need a "Process" but a "Thread". Pickle cannot dump a tkinter object, that's why you have this error. PS. If you don't need to parallelize heavy calculations, you should use threads – dgan Dec 12 '17 at 00:03
  • @dgan Originally I tried using threading. I couldn't get the behavior I wanted though. After the thread finished it kept running. When I used .join() to stop it, it would block the main thread. I tried calls to .after(). I tried for hours and read everything on stack I could find on the issue and never got it work. The whole point is to keep the GUI responsive. – twegner Dec 12 '17 at 00:09
  • I'm now using multiprocessing because I do not need communication between processes (shared resources). But it sounds like this is a no go according to your comment. – twegner Dec 12 '17 at 00:09
  • I persist to affirm that it's definitly threads that you need. If you need that your thread dies after the parent dies, then you set it as `deamon=True`. Joining the thread is blocking, that's normal. Otherwise did you have a look at https://stackoverflow.com/questions/323972/is-there-any-way-to-kill-a-thread-in-python ? Also, you can `return` as usual from thread, in order to leave it – dgan Dec 12 '17 at 00:13
  • 2
    Multiprocessing is a perfectly reasonable solution. You just can't reference any tkinter objects from the other process. Instead, have the process put items on a queue that the GUI thread can act upon. – Bryan Oakley Dec 12 '17 at 01:37

1 Answers1

1

Instead of using the multiprocessing module, which calls Popen, try using Popen directly (I dont know if this will work, but its worth a shot). This method shouldnt cause the call to pickle to happen.

In a separate file named worker.py

import time
import F06
import sys
import argparse

if sys.argc == 0:
    sys.exit(1)

for path in sys.argv:
    print('worker is processing:={}'.format(path))
    reader = F06.Reader()
    reader.read_sol_106(path)
    time.sleep(0.5)
    print('process complete')

Then in the main file.

from subprocess import Popen, PIPE

if __name__ == '__main__':

    # make a list of files to read
    cmd = [
        'python',
        'worker.py',
        'Results/1201301__SOL106.f06',
        'Results/1201302__SOL106.f06',
        'Results/1201303__SOL106.f06',
    ]
    p = Popen(cmd, stderr=PIPE, stdout=PIPE)
    out, err = p.communicate()
    print('main thread finished')
sterling baldwin
  • 181
  • 1
  • 3