6

I have the following code

my_func1()
my_func2()
my_func3()
my_func4()
my_func5()

Is it possible to calculate the data for the functions at the same time, instead of one after another?

PiccolMan
  • 4,854
  • 12
  • 35
  • 53

4 Answers4

15

My favorite way is using concurrent.futures which is a standard Python library (version 3.2 and above or available as a separate package for Python 2.7):

from concurrent.futures import ThreadPoolExecutor

executors_list = []

with ThreadPoolExecutor(max_workers=5) as executor:
    executors_list.append(executor.submit(my_func1, arg1, arg2))
    executors_list.append(executor.submit(my_func2, arg1, arg2))
    executors_list.append(executor.submit(my_func3, arg1, arg2))

for x in executors_list:
    print(x.result())

This will run concurrently my_func1, my_func2 and my_func3, passing arg1 and arg2 to each one. It will then print sequentially all the results as soon as they're available.

Giancarlo Sportelli
  • 1,219
  • 1
  • 17
  • 20
  • this does not rise exception when occured, is there a way to have it? – Nick Farsi May 06 '20 at 22:45
  • The exception is caught during execution and then re-raised when you iterate over the results. If you need to raise it before calling result() you'll have to catch it yourself and implement some form of inter-thread callback. – Giancarlo Sportelli May 07 '20 at 00:29
6

you can use multiprocessing or threading threading in python doesnt actually run in parallel but allow you to run the functions at the same time (and python will iterate them, doing a few lines from each at a time)

with multiprocessing they WILL run in parallel (assuming you have multiple cpu cores) but they wont share memory. here is a sample with multiprocessing

from multiprocessing import Process
p = Process(target=myfunc1)
p.start()
p2 = Process(target=myfunc2)
p2.start()
# and so on
p.join()
p2.join()
# the join means wait untill it finished

you can read more about it here:

https://docs.python.org/2/library/multiprocessing.html

https://wiki.python.org/moin/GlobalInterpreterLock

DorElias
  • 2,243
  • 15
  • 18
  • This definitely calculates my data a lot faster, but for some reason I am not able to display my data when it is calculated, if you know what I mean. – PiccolMan Sep 04 '15 at 20:51
  • I think my displaying function is calling itself before the multiprocess is finished. Is there a way to call my display function after the multiprocess functions are finished? – PiccolMan Sep 04 '15 at 21:02
  • yes, if you call it after you called p.join() on all your processes. join means it will wait untill it is finished, so you can wait untill they are all finish and only than call your display function – DorElias Sep 04 '15 at 21:05
3
from multiprocessing import Process
from time import sleep

def f(name):
    print 'hello', name
    sleep(1)

Consider above:

if you were to do this:

f('bob')  #start
f('alice') #wait until bob's done
f('jack') #wait until alice is done
f('cake') #wait until jack is done
f('paul') #wait until cake is done
print 'done'

you will end up waiting for 5 seconds, before you see done

however, if you utilize multiprocessing, you can spawn multiple processes to run the function at the same time.

Process(target=f, args=('bob',)).start() #start now
Process(target=f, args=('alice',)).start() #start now
Process(target=f, args=('jack',)).start() #start now
Process(target=f, args=('cake',)).start() #start now
Process(target=f, args=('paul',)).start() #start now
print 'done' #start now
taesu
  • 4,482
  • 4
  • 23
  • 41
2
import thread


thread.start_new_thread(my_func1, ())
thread.start_new_thread(my_func2, ())
thread.start_new_thread(my_func3, ())
thread.start_new_thread(my_func4, ())
thread.start_new_thread(my_func5, ())

You can thread functions like that

Yurippenet
  • 234
  • 1
  • 7