-1

I have a function that takes a "point" as an input and through an algorithm "minimizes" it (basically an optimizer of sorts).

Now I have a list of points to be minimized, but doing it serially through a loop wastes a lot of time where the program is just waiting for an external program to finish, so I need to a way to send more than one point at a time.

The way I've been doing it so far was to separate the optimization function to a different file and then call the function via a system call from the main program, i.e. something like: os.system('python3 filename.py ') and then check the output folder every few seconds to see if one finished so that it can be removed from the queue and allow the next point to start.

This technically works but it's not very elegant to say the least, so I was wondering if there's a better way to do this.

To summarize I want something like this:

for point in point_list:
    while len(queue) < MAX_QUEUE_LENGTH:
        wait until a space opens up
    add point to queue

This way if MAX_QUEUE_LENGTH = 4 there should always be 4 point running in parallel, and when one is finished the next point from the list would start.

Matan Ben Moshe
  • 41
  • 4
  • 4
    Possible duplicate of [Python: How can I run python functions in parallel?](https://stackoverflow.com/questions/7207309/python-how-can-i-run-python-functions-in-parallel) – fuglede Oct 06 '19 at 06:56
  • Seems like you want "multithreading" or even "multiprocessing". Just use these as search terms to find some info. – Ulrich Eckhardt Oct 06 '19 at 07:24

1 Answers1

1

See https://docs.python.org/3.7/library/multiprocessing.html

Something like this (The Point should be defined in your code)

from multiprocessing import Pool

def minimize_point(point):
    # do mnimize the point

if __name__ == '__main__':
    pool = Pool(5)
    print(pool.map(minimize_point, [Point(1,3), Point(2,5), Point(3,5)]))
balderman
  • 22,927
  • 7
  • 34
  • 52
  • So to make sure I understand this, this will always run 5 points in parallel until the list in the second argument of pool.map is done? – Matan Ben Moshe Oct 06 '19 at 07:14
  • 1
    5 is the size of the pool (and it is used as an example only). The points array will be divided between the 5 process and they will run at the same time. – balderman Oct 06 '19 at 07:17
  • Thanks, this worked. One more question: Is there a way to print from one of the processes in the pool to the terminal? or have the output from the pool go to the main program? – Matan Ben Moshe Oct 06 '19 at 07:41
  • 1
    You can return any thing you want from ` minimize_point`. The data you return can be "seen" by the main program – balderman Oct 06 '19 at 07:46