how to use asyncio and multiprocess.map to get data

Question

I am using asyncio to get the webpage.

However, the speed is not very high.

Here is the code:

async def get_page_url(url, session):
    res = await session.request(method="GET", url=url)
    return await res.text()

async def main_get_page_url(urls):      
    async with ClientSession() as session:
    tasks = [get_page_url(url, session) for province, url in urls]
    result = await asyncio.gather(*tasks)
    return result

if __name__ == '__main__':
    urls = ['http://www.cnn.com', 'http://www.bbc.com']
    loop = asyncio.ProactorEventLoop()
    asyncio.set_event_loop(loop)
    loop = asyncio.get_event_loop()
    df = loop.run_until_complete(main_get_page_url(urls))

I want to use multiprocessing.pool and map to increase the speed.

I have searched the web but cannot find any good method.

How to modify the code?

Any other better approach?

Thank you very much

What is taking time? What do you expect to speed up by using multiprocessing? — JohanL, Mar 29 '19 at 14:43
It's just an example. How to combine asyncio and multiprocessing. Any methods? — Chan, Mar 29 '19 at 16:14
It's a bad example, where you will gain nothing. Also, nowhere in your question do you say that this is only an example. To combine asyncio and multiprocessing is as simple as run the eval loop in one (or several of your processes). — JohanL, Mar 29 '19 at 22:31

score 1 · Accepted Answer · answered Mar 29 '19 at 17:01

You won't achieve anything using more processes. 99% of script execution time takes network I/O which you already handle using asyncio.gather. Only 1% of time takes CPU. Optimizing it already isn't worth investing time and increasing code complexity. If fact cost of spawning multiple processes may instead slowdown your script.

If you think your code runs slow than it should you should find a bottleneck in the first place and try to optimize it. Can't help you much more without fully reproducible example with actual code, urls and time measurements.

Disclaimer:

99% of script execution time takes network I/O

It's very rough approximation, but it's fair, take a look at this answer and especially at results at the very end of it.

how to use asyncio and multiprocess.map to get data

1 Answers1