Process a function on different arguments in parallel in Python

Question

This is my simple code where I want to run printRange() in parallel:

def printRange(lrange):
    print ("First is " + str(lrange[0]) + " and last is " + str(lrange[1]))


def runInParallel():
    ranges = [[0, 10], [10, 20], [20, 30]]
    // Call printRange in parallel with each sublist of ranges given as argument

My question is different from this SO question as here, the each process is hardcoded, started and finally joined. I want to run printRange() in parallel with say 100 other printRange() worker functions. Hardcoding each time is not feasible. How could this be done?

Parallelization is not panacea. There is overhead involved that is likely more expensive than this function, and because output is involved, there may be race conditions. You certainly can't expect the output to appear in the order implied by `ranges`, and in general you can't even assume that all the output from a single process will be written as a single unit. — chepner, Feb 20 '19 at 12:58

score 11 · Accepted Answer · answered Feb 20 '19 at 12:58

Using multiprocessing

from multiprocessing import Pool


def print_range(lrange):
    print('First is {} and last is {}'.format(lrange[0], lrange[1]))


def run_in_parallel():
    ranges = [[0, 10], [10, 20], [20, 30]]
    pool = Pool(processes=len(ranges))
    pool.map(print_range, ranges)


if __name__ == '__main__':
    run_in_parallel()

Output:

First is 0 and last is 10
First is 10 and last is 20
First is 20 and last is 30

score 1 · Answer 2 · answered Feb 20 '19 at 12:47

1

Something like this?

import threading

def printRange(lrange):
    print ("First is " + str(lrange[0]) + " and last is " + str(lrange[1]))

def runInParallel():
    ranges = [[0, 10], [10, 20], [20, 30]]
    for i in ranges:
        t = threading.Thread(target=printRange, args = [i])
        t.start()

runInParallel()

answered Feb 20 '19 at 12:47

Stef van der Zon

633
4
13

1

Threading is not parallel (although it gives the same headaches). Use multiprocessing instead. And you might want to add a join at the end – BlackBear Feb 20 '19 at 12:48
I thought the only difference was that multiprocessing uses seperate memory space. Threading is also parallel. Easy example: use 5 threads on a time.sleep(1) code and it will take 1 second total. – Stef van der Zon Feb 20 '19 at 12:51
Yes because the threads are sleeping :) try having the threads do some actual work, and in the meantime read about python's global interpreter lock: https://realpython.com/python-gil/ – BlackBear Feb 20 '19 at 12:57

score 0 · Answer 3 · answered Aug 23 '21 at 07:20

Using async/await for comparison

from asyncio import BaseEventLoop, get_event_loop, gather
from typing import List

async def print_range(lrange: List[int] = []):
  print('First is {} and last is {}'.format(lrange[0], lrange[1]))

def main():
  loop: BaseEventLoop = get_event_loop()
  lranges: list   = [[0, 10], [10, 20], [20, 30]]
  loop.run_until_complete(gather(*(print_range(_) for _ in lranges)))

if __name__ == '__main__':
  main()

Process a function on different arguments in parallel in Python

3 Answers3