4

Is there a way to calculate the max value for n_jobs given that my system CPU properties are known?

Here is the computation being performed:

classifier = KerasClassifier(build_fn=buildANN, batch_size=10, epochs=100)
accuracy = cross_val_score(estimator=classifier, X=X_train, y=y_train, cv=10, n_jobs=-1)

When I put n_jobs=-1, the process gets stuck and freezes on the python console. I'm running the application in Spyder IDE. When I put n_jobs=1, 2, .. etc then it will run but at a low speed. How do I calculate the max value for n_jobs as per my system CPU ? Is the quantity referring to number of CPU cores or number of parallel CPU's?

Rahul Raj
  • 3,197
  • 5
  • 35
  • 55
  • In 99% of all cases, your system is freezing due to memory-problems as every job will need it's own copy of the data (and there might be more). With enough memory, even 10000 jobs should work on single-core; maybe slow, but not freezing anything! – sascha May 15 '18 at 22:05
  • How much difference are you getting gor n_jobs =1 and 2? If its significant, you can try increasing the value until your system is stable? – Vivek Kumar May 16 '18 at 00:53
  • Yes, I can increase **n_jobs** and check the performance. But instead of checking 1000 times (for example), I would like to calculate the best value for **n_jobs** if **n_job=-1** not applicable. – Rahul Raj May 16 '18 at 05:23
  • And the only way to calculate that is to try each value. Otherwise you will have to delve into your data size, how much size each data type is taking, dig into the scikit code to determine which data is being copied to each thread and which data is common for all, how much size the thread is taking, etc – Vivek Kumar May 16 '18 at 07:58
  • @VivekKumar can one run with a single job, check the memory usage and use that as an estimate per thread? Then one could estimate `n_jobs` as `available_memory/memory_per_thread` – Mischa Lisovyi May 16 '18 at 11:25
  • Thats not how it will work if there are shared objects between the processes. – Vivek Kumar May 16 '18 at 11:48
  • you can check here, for the [answer](https://stackoverflow.com/a/50186510/7635458) – James Hermanto Nov 02 '20 at 16:08

0 Answers0