Tensorflow: Passing a session to a python multiprocess

Question

I'm using tensorflow to preprocess some large images. I was having a problem where the memory was rapidly collapsing. I turned to use multiprocessing in python so the memory would free up entirely whenever I want.

The thing is, I'm using python's multiprocess queues and for some reason unknown I can't pass my tensorflow session from my parent process to the children. Using some advanced debugging techniques (i.e. printing something every few lines) I noticed that python just goes idle inside the line where I make use of the session, it doesn't throw an error message.

My code looks something like this:

def subprocess(some_image, sess, q):
    with sess.as_default():
        # ... use sess and q ...
        print "All good and well" #This is printed
        some_image.eval() #Nothing happens here in console
        print "Still all good and well" #This is not printed

if __name__ == '__main__':
    # ... some initial operations ...
    some_image = read_some_image()

    sess = tf.Session()

    q = Queue()
    q.put(something)
    p = Process(target=subprocess, args=(some_image, sess, q))
    p.start()
    p.join()

What could be the problem? Many thanks!

Did you ever figure out how to run several sessions in parallel with the multiprocessing package? — MrRed, Apr 13 '16 at 23:54
If I remember correctly I was able to do what you want with multiprocessing pools. What I wanted was to have a single session shared accross multiple function calls, but I was not able to do that. In the end I opted for other methods to keep memory usage low — mathetes, Apr 14 '16 at 16:49
i realize this is pretty old, but I'm dealing with a similar issue. @mathetes you said you're able to pass sessions from the parent process into other child processes with multiprocessing pools? — cmed123, Dec 27 '19 at 20:15
sorry @cmed123 I don't remember the details anymore. I just remember that in the end I didn't push on this much farther, instead I found other ways to optimize performace. — mathetes, Jan 02 '20 at 12:27

score 3 · Accepted Answer · answered Jan 20 '16 at 12:41

3

I don't think you can share "state" as in the tf.Session() between processes like that. I would think that each process needed it's own session.

answered Jan 20 '16 at 12:41

svrist

7,042
7
44
67

At first look it seems so. My script is running very slow, I'm not sure if creating tensorflow sessions so often impacts performance or not – mathetes Jan 20 '16 at 12:52
Wouldn't you use the pool feature of Multiprocessing then? – svrist Jan 20 '16 at 12:58
I'll give it a try! I'm actually a novice in this python thing – mathetes Jan 20 '16 at 13:03
1

Indeed. You can't share the session like that. You have to dump the graph and reload it in a new Session. – fabrizioM Jan 20 '16 at 15:59

score -2 · Answer 2 · answered Feb 18 '17 at 02:23

-2

All you need is distributed tensorflow.

Create the graph and session in the parent process. Place some of the operators(especially, variables) to workers while constructing graph.
Create child processes, and run them

answered Feb 18 '17 at 02:23

Changming Sun

857
2
7
19

1

This answer is unclear, not giving details how to do that. I don't even think this is an answer. – imbr Apr 18 '18 at 23:14

Tensorflow: Passing a session to a python multiprocess

2 Answers2

Linked