3

While using keras I found that I couldn't use multiprocessing.Pool. After some troubleshooting I think importing keras is the source of the problem and have created a simple example of this. 

import keras
from multiprocessing import Pool

def foo(q,y):
    print("In foo")
    return q,y
def test(a, b):
    x = []
    if __name__ == '__main__':
        p = Pool(5)
        print("Starting")
        x = p.starmap(foo, [[a,2],[b,4]])
        print("Finished")
        p.close()
        p.join()
    print(x)

if __name__ == '__main__':
    test(1,3)

Output

> Starting

When run it outputs "Starting" then hangs. If I remove the keras import it runs fine and outputs [(1, 2), (3, 4)] as expected. Any idea how I can resolve this issue or what might be causing it? I don't fully understand how multiprocessing works with python yet. Thanks!

I am using anaconda and spyder for my code.

Tristhal
  • 110
  • 8
  • Suggest you a) structure your code as described in the multiprocessing docs https://docs.python.org/2/library/multiprocessing.html i.e. protect the last two lines by `if __name__ == ‘__main__’` and therefore I don’t think it makes much sense/adds value for the test method to call the foo method, and b) add some more debugging print statements - e.g. in foo and in test - to add more granualrity to where the code hangs. – DisappointedByUnaccountableMod May 09 '18 at 21:57
  • I made the changes you suggested. It didn't change the output however it should be noted that when the import is removed and it works "In foo" is never printed anyways. Thanks. – Tristhal May 09 '18 at 22:13
  • You added some more prints but didn’t move the `if __name__` protection: this matters, each of the subprocesses reads this same .py file, so follow the documentation. Follow the documentation. Follow the documentation. Got it? (May still not explain the problem with keras, but building on shaky foundations is - how shall I put it - shakier) – DisappointedByUnaccountableMod May 09 '18 at 22:14
  • I did change it i just hadn't updated it there. Its updated now. – Tristhal May 09 '18 at 22:14
  • Difficult to telepath that from here. – DisappointedByUnaccountableMod May 09 '18 at 22:15
  • How does foo get it’s ‘self’ as its not a class method. Humour me, try removing the class. Literally, delete the class definition line and outdent the two methods and call test() and foo(). – DisappointedByUnaccountableMod May 09 '18 at 22:18
  • The result is the same. It works without the import but not with. – Tristhal May 09 '18 at 22:22

1 Answers1

0

Here is a possible solution to this problem that worked for me since the child processes don't require keras to be imported.

if __name__ != '__mp_main__': #This line solves the problem
    import keras
from multiprocessing import Pool

def foo(q,y):
    print("In foo")
    return q,y
def test(a, b):
    x = []
    if __name__ == '__main__':
        p = Pool(5)
        print("Starting")
        x = p.starmap(foo, [[a,2],[b,4]])
        print("Finished")
        p.close()
        p.join()
    print(x)

if __name__ == '__main__':
    test(1,3)
    print(keras.backend)

How I found this solution was very weird. I ran the original code and it hung as normal. Then even as it was hung I commented out the import line and saved the file for the next test. As soon as I did this the code finished executing somehow. I tested this several times to make sure it was no fluke. How can editing the file of an already run program affect its operation? I then figured the if __name__ == '__main__' might be similar to if(fork() == 0) in c so I put it around the import so the children processes would not run it. Just as if I was manually commenting it out and saving during the run time. I tested it and it works.

Edit: If you use if __name__ == __main__ then inheritance gets broken since they aren't main it seems. mp_main is the name of the pool workers and this works as well

Tristhal
  • 110
  • 8