1

Hello my first post was yesterday working on basic understanding of asyncio and now I have another issue.

First post Converting concurrent futures to Asyncio python3.7

I ran this code two times once with using the asyncio function and one with pure threading.

import asyncio
import requests
import multiprocessing
from concurrent import futures


async def poll_data(data):
    response = requests.get('https://breadcrumbscollector.tech/feed/')
    print(f'Got data of length: {len(response.content)} in just {response.elapsed}')


async def start(loops):
    if loops:
        await loop.run_in_executor(loops)


def start_loop(data):
    loops = []
    loop = asyncio.new_event_loop()
    for foo in data:
        task = loop.create_task(poll_data(foo))
        loops.append(task)
    if loops:
        loop.run_until_complete(asyncio.wait(loops))


def poll_data_1(data):
    response = requests.get('https://breadcrumbscollector.tech/feed/')
    print(f'Got data of length: {len(response.content)} in just {response.elapsed}')


data =range(10)
CPUS = multiprocessing.cpu_count()
# asyncio
#with multiprocessing.Pool(processes=CPUS, maxtasksperchild=1) as pool: 
#    pool.imap_unordered(start_loop(data), 1) 
#    pool.close()
#    pool.join()

# threading
max_workers = 10
concurrent = futures.ThreadPoolExecutor(max_workers)


with concurrent as ex:
    ex.map(poll_data_1, data)

Results are as follows:

Asyncio + Multiprocessing
$ time python3 test.py
Got data of length: 160590 in just 0:00:01.036489
Got data of length: 160590 in just 0:00:00.947815
Got data of length: 160590 in just 0:00:00.557941
Got data of length: 160590 in just 0:00:00.573095
Got data of length: 160590 in just 0:00:00.484566
Got data of length: 160590 in just 0:00:00.954783
Got data of length: 160590 in just 0:00:00.930594
Got data of length: 160590 in just 0:00:00.915454
Got data of length: 160590 in just 0:00:01.057445
Got data of length: 160590 in just 0:00:00.959989

real    0m11.576s
user    0m0.533s
sys     0m0.220s


Threading
$ time python3 test.py
Got data of length: 160590 in just 0:00:00.605912
Got data of length: 160590 in just 0:00:01.632352
Got data of length: 160590 in just 0:00:02.022035
Got data of length: 160590 in just 0:00:02.045181
Got data of length: 160590 in just 0:00:01.992517
Got data of length: 160590 in just 0:00:01.980496
Got data of length: 160590 in just 0:00:02.030631
Got data of length: 160590 in just 0:00:01.975410
Got data of length: 160590 in just 0:00:02.008469
Got data of length: 160590 in just 0:00:02.044983

real    0m4.449s
user    0m0.533s
sys     0m0.229s

I thought with multiprocessing and asyncio the result would be much quicker. Or am I doing something massively wrong here?

Aladine
  • 185
  • 3
  • 16
  • 2
    asyncio won't help you if you make synchronous http requests. You should use a library like aiohttp instead of requests. – dirn Jul 27 '19 at 00:45
  • 1
    A good hint is when a function like `poll_data` is defined as `async def`, but doesn't contain a single `await`. That means it's async in name only, but as soon as asyncio starts executing it, it will continue until the end, blocking the event loop (and other coroutines) in the process. A correct asyncio corroutine `await`s anything that might block, and asyncio-enabled libraries make their API entry points awaitable. – user4815162342 Jul 27 '19 at 07:21
  • 2
    `asyncio` is useful when you use from the awaitable functions, `request` is not an awaitable function, instead of that `aiohttp` is. And [here is the difference between the concurrency methods](https://stackoverflow.com/questions/27435284/multiprocessing-vs-multithreading-vs-asyncio-in-python-3-4/52498068#52498068). – Benyamin Jafari Jul 27 '19 at 11:13

0 Answers0