1

Even after setting tf.config.threading.set_inter_op_parallelism_threads(1) and tf.config.threading.set_intra_op_parallelism_threads(1) Keras with Tensorflow CPU (running a simple CNN model fit) on a linux machine is creating too many threads. Whatever I try it seems to be creating 94 threads while going through the fitting epochs. Have tried playing with tf.compat.v1.ConfigProto settings but nothing helps. How do I limit the number of threads?

csankar69
  • 657
  • 1
  • 6
  • 13
  • 94 threads crashes your machine? I suggest it's not the number of threads that's responsible... – Mitch Wheat Jun 01 '20 at 23:09
  • 3
    My question is why is TF created so many threads even though inter & intra threads are set to 1? – csankar69 Jun 03 '20 at 00:33
  • I don't know the answer to your question, but, the x86_64 Linux kernel can handle a maximum of 4096 Processor threads in a single system image. 94 threads doesn't seem that many....Are you sure you have a problem that needs solving? – Mitch Wheat Jun 03 '20 at 05:37
  • 2
    That is probably a limit of the OS/kernel, the total threads for all users/processes - but regardless of the limit, there are only so many cores. You can spawn many threads without issue, if they are not doing anything, but once they start working, the load will go up. That is the issue I am having - have 40 physical cores (80 logical), regardless of threads. Can easily overwhelm the machine as threads surpass the cores. And the machine is being shared with other users who reserve cores for their jobs. In the end it is slowing up all jobs of all users. – csankar69 Jun 03 '20 at 15:06
  • This is probably an XY question. You have some actual problem (maybe) and you think getting TF to create fewer threads will solve it, so you ask how to get TF to create fewer threads. Is the issue high load? If so, tell us about the conditions under which you experience high load and ask about how to reduce the load. (Maybe process priorities?) – David Schwartz Feb 16 '21 at 21:01

1 Answers1

0

This is why tensorflow created many threads.

Using the mentioned 2 types of parallelism (inter and intra) you have limited control over the number of threads generated by TensorFlow. The minimum number of threads that you can get by setting these two variables is N, where N is the number of cores on your cpu (I don't know if you use gpu).

intra_op_parallelism_threads = 1
inter_op_parallelism_threads = 1

Even by setting the environment variables OMP_NUM_THREADS and MKL_NUM_THREADS can't help in further reducing the number of threads.

The following discussions suggest that without changing the source code of TensorFlow, it is not possible to reduce the number threads below N.

fisakhan
  • 704
  • 1
  • 9
  • 27