6

How do I run several python commands in parallel in a python script ? As a simple example I have several sleep commands:

time.sleep(4)
time.sleep(6)
time.sleep(8)

I want all the above to be executed in parallel. I expect the control back when 8 seconds have passed (which is the max of all the sleeps above). The real commands will be different but want to get an idea with above.

In bash, I could have simple done:

sleep 4 &
pid1=$!
sleep 6 &
pid2=$!
sleep 8 &
pid3=$!
wait $pid1 $pid2 $pid3

Thanks.

A J
  • 119
  • 1
  • 3

3 Answers3

10

One simple example, using multiprocessing:

import multiprocessing as mp
import time

pool = mp.Pool(3)
results = pool.map(time.sleep, [4, 6, 8] )

This spawns separate processes instead of creating separate threads in the same process as demonstrated in the answer by Steven Rumbalski. multiprocessing sidesteps the GIL (in cpython) meaning that you can execute python code simultaneously (which you can't do in cpython with threading). However, it has the downside that the information you send to the other processes must be pickleable, and sharing state between processes is a bit more complicated as well (although I could be wrong about the last point -- I've never really used threading).

word of caution: Don't try this in the interactive interpreter

mgilson
  • 300,191
  • 65
  • 633
  • 696
  • why that 'word of caution'? I ran it on the interactive interpreter and it worked... – nephewtom Aug 08 '17 at 15:04
  • @nephewtom -- Because there can be issues doing that depending on the OS. IIRC, Windows in particular has problems if you try to run the above in an interactive interpreter. – mgilson Aug 08 '17 at 18:22
  • Ok. I ran it on Linux. Thanks for the clarification! – nephewtom Aug 08 '17 at 23:38
7

Take a look at the threading module. You could have something like:

import time,threading

def mySleep( sec ):
    time.sleep( sec )
t1 = threading.Thread( target=mySleep, args=(4,) )
t2 = threading.Thread( target=mySleep, args=(6,) )
t3 = threading.Thread( target=mySleep, args=(8,) )
t1.start()
t2.start()
t3.start()

# All threads running in parallel, now we wait
t1.join()
t2.join()
t3.join()
Graeme Perrow
  • 56,086
  • 21
  • 82
  • 121
  • Excellent. I didn't know about the `args` argument to `threading.Thread`, and have appropriated it for my answer. Note that you don't need `mySleep`. Just do `target=time.sleep`. – Steven Rumbalski Aug 23 '12 at 18:08
  • Duh - you're right. `target=time.sleep` would work just as well. – Graeme Perrow Aug 23 '12 at 18:09
  • Threads in the CPython implementation have one _major_ disadvantage: only _one_ thread at a time can be executing python bytecode. See: http://wiki.python.org/moin/GlobalInterpreterLock So if the main task of the read is to run bytecode (rather than doing I/O or using extensions like numpy that are not bound by this limitation) threads will not speed up your program. The `multiprocessing` module than mgilson mentions below does not have that limitation. – Roland Smith Aug 23 '12 at 19:16
5
from threading import Thread

threads = [Thread(target=time.sleep, args=(secs,)) for secs in (4,6,8)]
for t in threads: t.start()
for t in threads: t.join()
print 'all threads done!'
Steven Rumbalski
  • 44,786
  • 9
  • 89
  • 119