1

System: Debian 10 running in the Google Cloud Platform as a DLVM, n1-highmem-16 machine type.

I'm doing some heavy calculations in Python, and I'm using the multiprocessing module to use a lot more of my CPUs. I've noticed, while watching my calculations spin in htop, that the idle processors jump around fairly frequently. That is, sometimes CPU 0 will be idle, and then it will become busy. Sometime later it might become idle again. This leads me to believe that the active jobs are probably jumping around as well. Surely the constant CPU hopping is slowing down the calculations? Is there a way to prevent the constant CPU hopping? Or is that a bad idea?

Here is a MWE of Python code you can run that will exercise multiprocessing long enough to detect the kind of hopping I'm talking about:

import multiprocessing
import numpy as np
import time


def function_calculation():

    data = np.zeros(1000000)
    t1 = time.perf_counter()
    direction = 1
    while time.perf_counter() - t1 < 600:  # 10 minutes.
        data = data + direction
        direction = direction * (-1)


num_processors = multiprocessing.cpu_count() - 1  # leave one out for system stuff.
for proc in range(num_processors):
    p = multiprocessing.Process(target=function_calculation)
    p.start()
Adrian Keister
  • 842
  • 3
  • 15
  • 33
  • briefly, it might, but if you're seeing a lot of idle CPU, I'd address that first! if you're using NumPy, you may find that it will already parallelize or thread operations and you can avoid `multiprocessing`, but may need to tweak some env vars if it's not behaving for you! some reading at https://stackoverflow.com/questions/17053671/how-do-you-stop-numpy-from-multithreading – ti7 Jun 14 '22 at 22:10
  • My actual function that I'm computing is an extremely complicated algorithm that only uses NumPy incidentally in pandas. Not sure what you mean by "addressing that first". I parcel out my data to the processors I want to use, then let 'em rip. Some of them, of course, finish before others, both because the data sent to each proc is not identical, and because the CPUs don't process all at the same rate. – Adrian Keister Jun 14 '22 at 22:13
  • I always hold one processor back, by the way, as I do in the MWE, to do system stuff. Among other things, that makes my remote desktop connection into the DLVM much more stable. – Adrian Keister Jun 14 '22 at 22:14
  • ah, makes sense and excellent - I was gathering that you might have _many_ idle cores all the time, which could just be used to accelerate your task! – ti7 Jun 14 '22 at 22:18
  • Well, I certainly don't have a lot of idle cores when I first start the calculations. As more and more of them finish, naturally I have more and more idle cores. I am looking into a load-balancing code that a colleague of mine has developed, but that's still future. – Adrian Keister Jun 14 '22 at 22:20
  • You can try using [`nice`](https://en.wikipedia.org/wiki/Nice_%28Unix%29) to modify the scheduling priority. – a_guest Jun 14 '22 at 22:37
  • @a_guest Well, maybe. Would changing the niceness towards -20 make CPU hopping less frequent? Or eliminate it? And how would I do that in Python? Would I have to grab the process's PID and then set the niceness based on that? – Adrian Keister Jun 14 '22 at 23:55
  • @AdrianKeister Did you read the Wikipedia article? All your questions are basically answered there. How a specific niceness value is interpreted depends on the scheduler of your OS. When you say that CPU hopping occurs "frequently", how frequent is that precisely? How bad is the effect on the overall runtime of your program? Is it relevant at all? – a_guest Jun 15 '22 at 22:24
  • @a_guest I don't see that the wiki answers my questions at all. The niceness appears to have a lot more to do with a particular CPU's tasks that are given to it: how it prioritizes those tasks. It does not appear to have anything to do with a task getting re-assigned from one CPU to another. The questions you're asking are precisely the questions I'm asking in the OP! – Adrian Keister Jun 15 '22 at 22:31
  • hmm.. my understanding is that making a task specific to a CPU can be done with `taskset` [recent question on SU](https://superuser.com/a/1721067/553189), while the niceness is how "friendly" a task is in allowing other tasks to [preempt](https://en.wikipedia.org/wiki/Preemption_(computing)) it .. neither fully Answers the Question about whether it this affects the calculation speed (though niceness will affect how often it's preempted) .. though as-is, the Question may not be directly answerable, but I suspect is simply "not much compared to other factors" (nice would matter with N big tasks) – ti7 Jun 15 '22 at 22:56
  • @AdrianKeister *"Would changing the niceness towards -20 make CPU hopping less frequent?"* - Yes, as the process receives a higher CPU priority and thus will yield less often to other processes. The niceness value is attached to a process and is thus relevant for all CPUs. *"Or eliminate it?"* - No, how a specific niceness value is interpreted depends on the scheduler. *"And how would I do that in Python?"* - `nice -n -20 python path/to/main.py`. Alternatively, `renice -n -20 -p `. – a_guest Jun 16 '22 at 12:51
  • When a process "hops" from one CPU to another, it will resume with a cold cache, i.e. no cache data, branching information and other optimization heuristics. How problematic this is depends on how cache sensitive your program is. However, before going into that much details, a much more important question to ask is, how frequent does this CPU hopping happen for your program? I.e. what is the overall runtime (wall time) of your program and how often does each process hop? This will help in estimating whether it's a relevant effect or not. – a_guest Jun 16 '22 at 12:52

0 Answers0