0

I have a function in python which I need to execute it multiple times and at the same time (to save computational cost). The following is a simple example and I don't know how to call my function at the same time!

def f(x):
    return x

y1=f(x1)
y2=f(x2)

I need to execute y1 and y2 at the same time and wait until both finish and save the results. Thank you all in advance.

Mohammad
  • 57
  • 1
  • 5
  • 1
    See the accepted answer here: [running python functions in parallel](http://stackoverflow.com/questions/7207309/python-how-can-i-run-python-functions-in-parallel) – prashkr Jun 23 '16 at 12:03

2 Answers2

2

I would encourage you to read the https://docs.python.org/3/library/multiprocessing.html documentation, which has a great example. If you're using Python 3.x

from multiprocessing import Pool
def f(x):
    return x*x

if __name__ == '__main__':
    with Pool(5) as p:
        print(p.map(f, [1, 2, 3]))

If you're using python 2.7: https://docs.python.org/2.7/library/multiprocessing.html

from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == '__main__':
    p = Pool(5)
    print(p.map(f, [1, 2, 3]))
AbrahamB
  • 1,348
  • 8
  • 13
  • multiprocessing Pools are definitely the way to go, but PLEASE read all the warning notes in the docs! – Exelian Jun 23 '16 at 12:09
  • Thank you very much for your response. I'm new to python and when I run this code in IDLE, I get this error, while I should print out [1,4,9] Traceback (most recent call last): File "C:/Users/mohammad/Desktop/r.py", line 8, in with Pool(5) as p: AttributeError: __exit__ – Mohammad Jun 23 '16 at 14:00
  • If you're using python 2.x (like python 2.7), use the instructions here: https://docs.python.org/2.7/library/multiprocessing.html I've added this to the answer. – AbrahamB Jun 23 '16 at 14:05
  • Thank you very much @AbrahamB I'm wondering if number 5 shows the parallelization? all the 3 inputs will be executed at the same time? if one of them returns longer than the others, it doesnt crash? – Mohammad Jun 23 '16 at 14:32
  • @Mohammad, per: https://docs.python.org/2/library/multiprocessing.html#module-multiprocessing.pool: `processes is the number of worker processes to use.` – AbrahamB Jun 23 '16 at 14:47
  • @AbrahamB tnx again. In windows it is working fine, but when I run it in Linux, I think there is a bug for multiprocessing and nothing happens when I run the code! not even an error! – Mohammad Jun 23 '16 at 15:10
  • That's probably a large enough issue for a whole new question :) I am able to run it fine on my linux box, but there can be many differences such as python version, OS bits, etc. – AbrahamB Jun 23 '16 at 16:03
2

Another solution that might help you:

import Queue
import threading

# your function with a slight change:
def f(q,x):
    q.put(x)

inputs = [1,2,3,4]

# aggregate to a queue:
q = Queue.Queue()

for x in inputs:
    t = threading.Thread(target=f, args=(q,x))
    t.daemon = True
    t.start()

outputs = q.get()
print outputs

output: 1

Here q holds the return value of f.

In any case I suggest you try and read this.

Gal Dreiman
  • 3,969
  • 2
  • 21
  • 40
  • Thanks for your response. As I'm new to python, I don't know how to execute the file! As I understood, it should calls the f, 4 times in parallel and we should have 4 results in the output? – Mohammad Jun 23 '16 at 13:55
  • For every `q.get()` the return value is the output. – Gal Dreiman Jun 26 '16 at 06:14