11

I am training a LSTM model on a very huge dataset on my machine using Keras on Tensorflow backend. My machine have 16 cores. While training the model I noticed that the load in all the cores are below 40%.

I have gone through different sources looking for a solution and have tried providing the cores to use in the backend as 

config = tf.ConfigProto(device_count={"CPU": 16})
backend.tensorflow_backend.set_session(tf.Session(config=config))

Even after that the load is still the same.

Is this because the model is very small.? It is taking around 5 minutes for an epoch. If it uses full cores the speed can be improved.

How to tell Keras or Tensorflow to use the full available cores i.e 16 cores to train the model.??

I have went through these stackoverflow questions and tried the solutions mentioned there. It didn't help.

Limit number of cores used in Keras

Sreeram TP
  • 11,346
  • 7
  • 54
  • 108

1 Answers1

0

How are you training the model exactly? You might want to look into using model.fit_generator() but with a Keras Sequence object instead of a custom generator. This allows to safely use multiprocessing and will result in all cores being used.

You can checkout the Keras docs for an example.

sdcbr
  • 7,021
  • 3
  • 27
  • 44
  • How is the system/machine affected if all cores are used for training? In my case I don't have a separate system for training so I have to train it on my personal machine. While training, can I still use the system for regular stuff like browsing&coding? Whyever I can only use 1 of my 2 cores atm but as I have to work besides while training this is okish for me. On the other hand, if I still can work while training on both cores, that would be better :) – Ben Oct 11 '19 at 10:12