Should I set OMP_NUM_THREADS=1 on Machine Learning Server?

Question

I have an Machine Learning (ML) Linux server setup on which multiple machine learning algorithms runs in parallel. The ML system is Python based and has most of the standard ML Frameworks and libraries like TensorFlow, Sklearn, Numpy, Dask, Pandas etc. installed.

When algorithms runs simultaneously, I frequently get Resource temporarily unavailable error and it asks me to decrease value of OMP_NUM_THREADS.

I was wondering whether I should reduce it, as I assume it will disable multithreading and further slow down algorithms. What is best way to tackle this issue?

Answer 1

I don't think there is a single answer to this question. It is very dependant on what you are trying to achieve and there are several settings you probably want to play with:

• OMP_NUM_THREADS - This environment variable sets the maximum number of threads to use for OpenMP parallel regions (source)

Some TF specific configurations (source)

• inter_op_parallelism_threads - Nodes that perform blocking operations are enqueued on a pool of inter_op_parallelism_threads available in each process.
• intra_op_parallelism_threads - The execution of an individual op (for some op types) can be parallelized on a pool of intra_op_parallelism_threads.

Additionally there are settings for the level of parallism of pytorch and opencv (if you use them).

In my experience if you are running multiple models at the same time you really want to understand what is the best setting that will work for you. Default behaviour would typically assume the system executes just the current model and may be suboptimal for what you are trying to do.