1
    from multiprocessing import Pool
    pending_integrations = DataService().get_pending_integrations(args.integration)
            for integration in pending_integrations:
                integration_id = integration.get('id')
                platform = integration.get('platform')
                print(integration_id , platform)
                await runner.run(integration_id, platform)

this is the output of pending_integrations: [{'id': 1, 'platform': 'one'}, {'id': 2, 'platform': 'two'}, {'id': 3, 'platform': 'three'}]

my requirement is to execute the for loop in parallel. I tried to run this using multiprocessing as below

from multiprocessing import Pool
pending_integrations = DataService().get_pending_integrations(args.integration)
if __name__ == '__main__':
    pool = Pool(3)
    pool.starmap(await runner.run(integration_id, platform), pending_integrations)

this is what run method takes as arguments: async def run(self, integration, platform)

this is the error i am getting: run() missing 2 required positional arguments: 'integration' and 'platform'

karthik kancharla
  • 11
  • 3
  • You can’t mix asyncio and multiprocessing that way. If `run` is defined using `async def`, why not just use asyncio to run your code? – dirn Jul 04 '21 at 13:24
  • so should i remove async before run function and call it as a normal function? – karthik kancharla Jul 04 '21 at 13:58
  • Unless `run` awaits something. Then you may also need to change the body of it (and possibly dependencies). – dirn Jul 04 '21 at 14:00
  • is there any way i can run the code in parallel using asyncio in my case? – karthik kancharla Jul 04 '21 at 14:08
  • asyncio uses cooperative multitasking, which may or may not satisfy your needs. Without knowing what `run` does its impossible to say. If it does IO, [the docs cover various ways to run things concurrently based on your exact needs](https://docs.python.org/3/library/asyncio-task.html#running-tasks-concurrently). If `run` is mostly CPU-bound then asyncio won’t help much. – dirn Jul 04 '21 at 14:16
  • there are three other functions which gets called based on the platform. I am using pyppeteer in those functions to get the cookies. that's the reason i am using async to my run function. – karthik kancharla Jul 04 '21 at 14:36

0 Answers0