I want to use multiprocessing to spread work across a system's multiple cores. As part of the work, they will run subprocess.call(..., shell=True). What happens when they do that? Does the subprocess fork stay on that core?
Asked
Active
Viewed 760 times
1
user592419
- 5,103
- 9
- 42
- 67
-
Did you try it? Usually, new processes are scheduled by the OS. – poke Sep 08 '15 at 17:26
-
Probably not. You usually have to explicitly bind processes to core. The method to do this is OS-dependent. – Cong Ma Sep 08 '15 at 17:27
1 Answers
2
If the main work is done in the child processes created using subprocess module then you don't need multiprocessing to spread the work across multiple CPU cores. See Python threading multiple bash subprocesses?
What happens when they do that?
subprocess.call() runs an external command and waits for it to complete. It doesn't matter whether it is started inside a worker process created by multiprocessing module or not.
Does the subprocess fork stay on that core?
If you need it; you should set CPU affinity explicitly. psutil provides a portable way to set/get CPU affinity for a process.
If you use numpy then it may affect the cpu affinity. See Why does multiprocessing use only a single core after I import numpy?
