2

I have following problem. I'm using a Tensorflow Keras model to evaluate continuous sensor data. My input for my model consists of 15 sensor data frames. Because the function model.predict() takes near 1 second I wanted to execute this function asynchronous so that I can collect the next data frames in this time period. To accomplish this I created a Pool with the multiprocessing libary and a function to for model.predict. My code looks something like this:

def predictData(data): 
   return model.predict(data)

global model
model = tf.keras.models.load_model("Network.h5")
model._make_predict_function()

p = Pool(processes = 4)
...
res = p.apply_async(predictData, ([[iinput]],))
print(res.get(timeout = 10))

Now I always get a timeout-error when calling predictData(). It seems like model.predict() is not working right. What am I making wrong?

rexcoder
  • 21
  • 1
  • 3
  • The computation graph that is built in the tensorflow backend exists outside of the python framework. Using multiprocessing in this way does not build multiple copies of the graph. You still only have one copy of the model and are trying to send 4 streams of data at it all at once. – James Oct 18 '18 at 10:31
  • James is right, consider running the prediction on the foreground process and run a thread/process in the background to collect your next data frame(s). You can buffer multiple data frames are put them in the batch dimension of the input to the network – MohamedEzz Oct 18 '18 at 10:33
  • okey then collection the data via threads seems to be the right method. So in general it is not possible to run e.g. multiple prediction in multiple processes? – rexcoder Oct 18 '18 at 11:16

2 Answers2

1

The reason is that each process you spawn will require a new initialized version of your model which it uses to make predictions. Therefore you have to make sure you instantiate/load your model for every spawned process. This is defiantly not optimal.

This is a known caveat with multiprocessing machine learning training and/or inference. Some libraries come with multiprocessing features out-of-the-box and provide parallizable calls to their models. However, in most libraries once you want to do multiprocessing, you are on your own!

Make sure you instantiate your model once and then find a way to share that model across processes. One basic way to do that, is to serve your model as a flask service then make predictions against that service to your hearts content. Cheers!

moeabdol
  • 4,779
  • 6
  • 44
  • 43
0

It is possible to run multiple predictions in multiple concurrent python processes, only you have to build inside each independent process its own tensorflow computational graph and then call the keras.model.predict

Write a function which you will use with the multiprocessing module (with the Process or Pool class), within this function you should build your model, tensorflow graph and whatever you need, set all tensorflow and keras variables, then you can call the predict method on it, and then pipe the result back to your master process.

for example:

    def f(data):

          import tensorflow, keras

          configure your tensorflow and keras settings (e.g.  GPU/CPU usage)

          keras_model = build_your_keras_model()

          result = keras_model.predict(data)

          return result

    if __main__ = '__main__':

          p = Pool(processes = 4)

          res = p.apply_async(f, (data,))

          print(res.get(timeout = 10))
olivierjeanmarie
  • 9
  • 2