Run one function on different CPUs

Question

I have one machine with two CPUs, and each CPU has different number of cores. I have one function in my python code. How can I run this function on each of the CPUs?

In this case, I need to run function two times because I have two CPUs.

I want this because I want to compare the performance of different CPU. This can be part of code. Please let me know if the code it not written in correct way.

import multiprocessing

def my_function():
    print ("This Function needs high computation")
    # Add code of function

pool = multiprocessing.Pool()
jobs = []
for j in range(2): #how can I run function depends on the number of CPUs?
    p = multiprocessing.Process(target = my_function)
    jobs.append(p)
    p.start()

I have read many posts, but have not found a suitable answer for my problem.

Which OS are you using, I don't know the answer but think it will make a difference — JeffUK, Mar 28 '21 at 15:57
I'm pretty sure `multiprocessing.Process` runs functions in seperate cores already? — coderman1234, Mar 28 '21 at 15:58
This is not really a Python question; it's an OS question. You need to research how to run on a specified core for your operating system and platform. *Then* you can likely execute your script using the appropriate shell command. — Prune, Mar 28 '21 at 15:58
Does this answer your question? https://stackoverflow.com/questions/36172101/designate-specific-cpu-for-a-process-python-multiprocessing — Stefan Falk, Mar 28 '21 at 15:59

pcko1 · Answer 1 · 2022-02-14T11:20:51.507

The concurrent package handles the allocation of resources in an easy way, so that you don't have to specify any particular process/thread IDs, something that is OS-specific anyway.

If you want to run a function using either multiple processes or multiple threads, you can have a class that does it for you:

from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor
from typing import Generator

class ConcurrentExecutor:
    @staticmethod
    def _concurrent_execution(executor, func, values):
        with executor() as ex:
            if isinstance(values, Generator):
                return list(ex.map(lambda args: func(*args), values))
            return list(ex.map(func, values))

    @staticmethod
    def concurrent_process_execution(func, values):
        return ConcurrentExecutor._concurrent_execution(
            ProcessPoolExecutor, func, values,
        )

    @staticmethod
    def concurrent_thread_execution(func, values):
        return ConcurrentExecutor._concurrent_execution(
            ThreadPoolExecutor, func, values,
        )

Then you can execute any function with it, even with arguments. If it's a single argument-function:

from concurrency import ConcurrentExecutor as concex

# Single argument function that prints the input
def single_arg_func(arg):
    print(arg)

# Dummy list of 5 different input values
n_values = 5
arg_values = [x for x in range(n_values)]

# We want to run the function concurrently for each value in values
concex.concurrent_thread_execution(single_arg_func, arg_values)

Or with multiple arguments:

from concurrency import ConcurrentExecutor as concex

# Multi argument function that prints the input
def multi_arg_func(arg1, arg2):
    print(arg1, arg2)

# Dummy list of 5 different input values per argument
n_values = 5
arg1_values = [x for x in range(n_values)]
arg2_values = [2*x for x in range(n_values)]

# Create a generator of combinations of values for the 2 arguments
args_values = ((arg1_values[i], arg2_values[i]) for i in range(n_values))

# We want to run the function concurrently for each value combination
concex.concurrent_thread_execution(multi_arg_func, args_values)

Run one function on different CPUs

1 Answers1