Lowering process priority of multiprocessing.Pool on Windows

Question

I use multiprocessing.Pool() to parallelize some heavy Pandas processing but find that it is a bit too successful. My CPU usage goes to 100% and my entire computer becomes very unresponsive. Even the mouse becomes difficult to use.

I can change the process priority of my process with this code.

import psutil
p = psutil.Process(os.getpid())
p.nice = psutil.BELOW_NORMAL_PRIORITY_CLASS

However, when I look in Windows Task Manager I find that only the main python.exe process has been changed to below normal priority.

Is there a good way to reduce the priority of the pool processes?

Have you tried putting the snippet you show inside the worker function? — Lev Levitsky, Apr 14 '14 at 13:05
@acushner I've temporarily hacked it to use a pool size of 7 which has 'fixed' it. But I wonder if there's a better solution. — JasonEdinburgh, Apr 14 '14 at 13:55
@levitsky I expected the psutil code would be slow to run repeatedly , so I didnt try that. I will. If there was a way to execute it once only per process that would be better. I thought there might be a better way to do this, perhaps instructing the Pool to use lower priority processes. — JasonEdinburgh, Apr 14 '14 at 13:56

score 7 · Accepted Answer · answered Apr 17 '14 at 08:29

7

You can try setting priority of your process' children after you spawned them. Something like:

import psutil

# spawn children and/or launch process pool here

parent = psutil.Process()
parent.nice(psutil.BELOW_NORMAL_PRIORITY_CLASS)
for child in parent.children():
    child.nice(psutil.BELOW_NORMAL_PRIORITY_CLASS)

answered Apr 17 '14 at 08:29

Giampaolo Rodolà

12,488
6
68
60

YES ! thank you .. I had to modify it a bit. You might want to modify your answer for anyone else who finds this – JasonEdinburgh Apr 17 '14 at 10:37
parent = psutil.Process() parent.nice = psutil.BELOW_NORMAL_PRIORITY_CLASS for child in parent.get_children(): child.nice = psutil.BELOW_NORMAL_PRIORITY_CLASS – JasonEdinburgh Apr 17 '14 at 10:38
The other way I found to do this was with ProcessHacker2. http://processhacker.sourceforge.net/ – JasonEdinburgh Apr 17 '14 at 10:40
1

"child.nice = value" was deprecated in psutil 2.0.0. – Giampaolo Rodolà Apr 17 '14 at 13:55

score 2 · Answer 2 · answered Feb 09 '19 at 13:32

The same result as by using the answer by @Giampaolo Rodolà is achieved simply by setting the parent process priority before spawning the children:

import psutil

parent = psutil.Process()
parent.nice(psutil.BELOW_NORMAL_PRIORITY_CLASS)
# the rest of your code

The children processes will inherit the parent's priority. If, however, the parent is to be set to different priority than the children, then the code provided by @Giampaolo Rodolà is needed.

score 1 · Answer 3 · answered Jan 07 '20 at 08:15

The Python documentation states that when a pool is created you can specify the number of processes. If you don't, it will default to os.cpu_count. Consequently, you get the expected behavior that all the available logical cores are used. In turn, the computer becomes unresponsive.

It would probably be better to do something simpler by just controlling the number of processes created. A rule of thumb is to reserve 2 to 4 logical cores for interactive processing.

Also, the Python documentation states "This number [os.cpu_count()] is not equivalent to the number of CPUs the current process can use. The number of usable CPUs can be obtained with len(os.sched_getaffinity(0))"

There are several other details that need to be addressed. I have tried to capture them at this gist. All that you have to do is change LOGICAL_CORES_RESERVED_FOR_INTERACTIVE_PROCESSING for your particular use case.

Lowering process priority of multiprocessing.Pool on Windows

3 Answers3

Linked