0

Whould you prompt me easy solution, how to limit the number of CPU cores in Python 3.5. I check it by using:

import multiprocessing
print(multiprocessing.cpu_count())

When I run tensorflow with 1200 1st layer neurons (abslolutely identical dataset, Python and libraries version), it shows identical results on servers with 24 and 32 cores, but different - on my local machine with 12 cores (details - in my yesterday SO article1).

I know for sure, changing the CPU cores number is the key to solving the problem, since when I decrease 1st layer neurons, 3 machines show indentical results. Probably, when processing large amount of data, calculating process slightly depends on CPU power.

Alex Ivanov
  • 657
  • 1
  • 8
  • 17
  • 1
    https://stackoverflow.com/questions/55746872/how-to-limit-number-of-cpus-used-by-a-python-script-w-o-terminal-or-multiproces Does this work for your situation? – Axiumin_ Jun 03 '20 at 05:22
  • Axiumin, thank you. This solves the problem via os.environ. But my os.environ doesn't have the word 'thread' at all. It only has 'NUMBER_OF_PROCESSORS' (which = 12 in my local machine and 32 in server). I changed it in server from 32 to 12 but it didn't help. Results are still different. – Alex Ivanov Jun 03 '20 at 05:55
  • 1
    It seems like you need to configure these things in TensorFlow and not in python. Have you looked at tf.config.set_visible_devices? – Roy2012 Jun 03 '20 at 06:58
  • Roy2012, thank you. I totally agree, that it can be linked with tensorflow itself. As far as I can see, tf.config.set_visible_devices is linked with GPU tuning, but I don't have GPU. May be there is another parameter? E.g. I saw tf.config.threading.set_inter_op_parallelism_threads, but in both server and local machine it equals 0. – Alex Ivanov Jun 03 '20 at 09:20

1 Answers1

0

Solved the problem by adding in the very beginning of my code:

tf.config.threading.set_inter_op_parallelism_threads(1)
tf.config.threading.set_intra_op_parallelism_threads(1)

Roy2012 - thank you very much.

Alex Ivanov
  • 657
  • 1
  • 8
  • 17