How could I get share directory in multiprocess when pressing ctrl+c

Question

I'm using the multiprocessing module to do parallel processing in my program, I Want to get the share dict object between multi-processes, I could do it when multi-process is closed normally, but could not get it when pressing CTRL+C, How could i achieve my goal ? my code as follows

#!/usr/bin/python
from multiprocessing import Process, Manager, Pool
import os
import signal
import time

def init_worker():
    signal.signal(signal.SIGINT, signal.SIG_IGN)

def run_worker(i,my_d):
    print 'Work Started: %d %d' % (os.getpid(), i)
    for j in range(5):
        print j
        tmp1 = str(i) + str(j)
        my_d[tmp1] = j
        time.sleep(1)

def main():
    print "Initializng 3 workers"
    pool = Pool(3, init_worker)

    manager = Manager()
    my_d = manager.dict()
    try:
        for i in range(3):
            pool.apply_async(run_worker,args=(i,my_d))
        pool.close()
        pool.join()
        print my_d
# When process is closed normally, I could get the my_d successfully

    except KeyboardInterrupt:
        print "Caught KeyboardInterrupt, terminating workers"
        pool.terminate()
        pool.join()
        print my_d
#When process is closed by Ctrl+C, couldn't I get the my_d ?

if __name__ == "__main__":
    main()

Are you able to get `my_d` in the `except`? – nneonneo Sep 16 '12 at 05:56 — nneonneo, Sep 16 '12 at 05:56

score 0 · Answer 1 · edited May 23 '17 at 12:03

0

You need a shared dictionary, from the Manager-Object in multiprocessing.

See this similar question here (first answer): Python multiprocessing: How do I share a dict among multiple processes?

edited May 23 '17 at 12:03

Community

1
1

answered Sep 16 '12 at 11:38

Gregor

4,306
1
22
37

Thank yor for help, but my question is how to get share dict among multi-process when pressing ctrl +c, no matter use which way, however, thanks again – user1675167 Sep 17 '12 at 02:26

Roland Smith · Answer 2 · 2012-09-17T19:18:02.127

Look at the error that is produced when you interrupt the parent process:

Caught KeyboardInterrupt, terminating workers
<DictProxy object, typeid 'dict' at 0x801abe150; '__str__()' failed>

Try changing print "Caught KeyboardInterrupt, terminating workers" to print len(my_d) and you can see in detail what happens. Note that this is before you try and terminate/join the pool of workers:

Traceback (most recent call last):
  File "manager-test.py", line 39, in <module>
    main()
  File "manager-test.py", line 33, in main
    print len(my_d)
  File "<string>", line 2, in __len__
  File "/usr/local/lib/python2.7/multiprocessing/managers.py", line 755, in _callmethod
    self._connect()
  File "/usr/local/lib/python2.7/multiprocessing/managers.py", line 742, in _connect
    conn = self._Client(self._token.address, authkey=self._authkey)
  File "/usr/local/lib/python2.7/multiprocessing/connection.py", line 169, in Client
    c = SocketClient(address)
  File "/usr/local/lib/python2.7/multiprocessing/connection.py", line 293, in SocketClient
    s.connect(address)
  File "/usr/local/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 2] No such file or directory

When you interrupt the main program, the connection from the child processes to the manager will be broken. This leaves the manager (and the objects that it manages) in an unusable state. The socket connection from the manager to the child processes doesn't work anymore, so the proxy cannot get the data.

If you want to interrupt long-running processes without losing data, you should do it more gently, I think. Something like this:

import select
import sys

print 'Type q<enter> it you want to quit...'
while True:
    r, foo, bla = select.select([sys.stdin], [], [], 1)
    if len(r):
        what = sys.stdin.readline()
        if 'q' in what:
            print 'bye!'
            break;
    # E.g. check on the progress of your calculation here
# Close and join the pool here, and do other clean-up.

Thank you very much for further analyse, do you has any other ways to solove this problem? no need to conside Manager only. I could just get share ojbects among process when pressing CTRL+C. Thanks again — user1675167, Sep 17 '12 at 02:31
Using ctrl+c to close the program is like using an axe to open a door. It works, but afterwards the door is broken. If you want to interrupt long-running subprograms, you have to do it differently. I'll edit my answer with some pointers. — Roland Smith, Sep 17 '12 at 18:57

How could I get share directory in multiprocess when pressing ctrl+c

2 Answers2