-1

I want to call 4 methods at once so they run parallel-ly in Python. These methods make HTTP calls and do some basic operation like verify response. I want to call them at once so the time taken will be less. Say each method takes ~20min to run, I want all 4methods to return response in 20min and not 20*4 80min

It is important to note that the 4methods I'm trying to run in parallel are async functions. When I tried using ThreadPoolExecutor to run the 4methods in parallel I didn't see much difference in time taken. Example code - edited from @tomerar comment below

from concurrent.futures import ThreadPoolExecutor

async def foo_1():
    print("foo_1")

async def foo_2():
    print("foo_2")

async def foo_3():
    print("foo_3")

async def foo_4():
    print("foo_4")

with ThreadPoolExecutor() as executor:
  for foo in [await foo_1,await foo_2,await foo_3,await foo_4]:
    executor.submit(foo)

Looking for suggestions

heythere825
  • 9
  • 1
  • 1
    you seem to already know [multithreading](https://docs.python.org/3/library/threading.html) ... did you try it ? show us what you tried so far. – Ahmed AEK Jan 14 '23 at 20:09
  • 1
    Note: With `threading`, your four threads can await HTTP responses in parallel, but they cannot do any computation in parallel (Google for "Python" and "GIL".) If most of that 20 minutes is spent awaiting responses from remote servers, then `threading` will work for you. But, if a large part of that 20 minutes is spent "processing" the replies, then you may be better off using `multiprocessing`. – Solomon Slow Jan 14 '23 at 20:30

2 Answers2

1

You can use from concurrent.futures import ThreadPoolExecutor

from concurrent.futures import ThreadPoolExecutor

def foo_1():
    print("foo_1")

def foo_2():
    print("foo_2")

def foo_3():
    print("foo_3")

def foo_4():
    print("foo_4")

with ThreadPoolExecutor() as executor:
  for foo in [foo_1,foo_2,foo_3,foo_4]:
    executor.submit(foo)
tomerar
  • 805
  • 5
  • 10
0

You can use "multiprocessing" in python. it's so simple

from multiprocessing import Pool
pool = Pool()
result1 = pool.apply_async(solve1, [A])    # evaluate "solve1(A)" 
result2 = pool.apply_async(solve2, [B])    # evaluate "solve2(B)" 
answer1 = result1.get(timeout=10)
answer2 = result2.get(timeout=10)

you can see full details

tomerar
  • 805
  • 5
  • 10
behzad
  • 1
  • 3